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Executive Summary 


The Better Math Teaching Network (BMTN) is organized as a networked improvement community, which 
is a structured network of researchers and practitioners working together to address a common problem 
of practice, in this case, how to improve opportunities to increase student engagement in high school 
mathematics. During the 2017-18 school year, 41 teachers from all six New England states participated in 
the second official year of the network's operation. Teachers were selected from a pool of volunteers that 
applied to be part of the initiative. Participating teachers work in urban, suburban, and rural contexts and 
teach at least one Algebra | course to 9th grade students. Teachers committed to work collaboratively to 
make their teaching more student-centered using the improvement science approach. 


Chapter 1: BMTN Routines and Tools 


Routines and tools are often used by leaders in organizations to structure work practices and to catalyze 
and sustain change. In the Better Math Teaching Network, network leaders designed and refined routines 
and tools they intended would support both teachers’ use of improvement science and their engagement 
in the network. By intentionally and carefully designing these routines and tools, BMTN network leaders 
scaffolded teacher learning and supported teachers as they changed their practices. In this chapter we 
examine three routines and three tools that BMTN network leaders designed and implemented. We 
consider each routine and tool's designed purpose. We then discuss the benefits and challenges teachers 
experienced as they engaged in each routine or used each tool, as reflected in our data. Finally, we note 
possible design implications for the field. 
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Routines that were most effective involved regularity, collaboration, and learning. 


Building coherence and connections across routines and tools can streamline 
messages and learning. 


Balancing in-person and virtual meetings that occur regularly can support momentum, 
accountability, and engagement. 


* The ways that tools are designed can teach users and focus attention on particular 
constructs or principles. 


Simple tools work well for busy teachers, as does building in time for structured or 
guided reflection. 


Finding ways to capture and consolidate learning from inquiry cycles is a challenge. 


Chapter 2: Influence of Routines on the Collaborative Work of BMTN Teachers 


Better Math Teaching Network leaders designed and refined a series of tools and routines to 

support participant learning and engagement in BMTN. Three of these routines—often referred to as 
“participation structures” in networked improvement communities—organized the collaborative work 

of BMTN teachers: network meetings, PDSA meetings, and optional study group meetings. In this chapter 
we examine the importance of these routines for building and sustaining collaboration, an element 
critical to both learning and the acceleration of learning in networked improvement communities. Our 
social network analysis suggests that in general, collaboration among BMTN teachers happens almost 
exclusively within the formally structured interactions and not informally outside of these designed 
participation spaces. These findings highlight the importance of the design and use of formal participation 
structures because these leader-designed structures determine with whom and how intensely teachers 
collaborate. Given that participation structures largely shape the collaborative work that teachers do, 
network leaders must intentionally design them to achieve network goals. Participants also noted that the 
in-person whole network meetings were critical to building and maintaining trust, sparking motivation, 
and stretching their thinking. While these meetings are resource intensive, teachers see face-to-face time 
as essential to the work. 


Chapter 3: The Challenges of Improvement Science 


Learning and integrating improvement science into practice is the biggest challenge teachers face as 
they engage in the Better Math Teaching Network. In end-of-year interviews, 53% of returning teachers 
and 78% of teachers new to the network identified an aspect of improvement science as one of the 
most significant challenges they encountered as they engaged in the work. 


Specifically, they spoke about challenges with: 


* Identifying good change ideas to test 
¢ Using data and improvement to formally test changes 


* Integrating their improvement activities into their workflow (e.g., remembering to do it, 
fitting it into the flow of classroom instruction) 
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Once teachers become accustomed to engaging in inquiry cycles, some aspects of improvement science 
become less challenging for them. However, issues around data collection and measurement persist into 
teachers’ second and third years of participation in BMTN, indicating that this challenge is harder to solve 
than teachers learning to integrate inquiry cycles into their practice over the course of their first year. 


Teachers tend to run into three challenges as they work to design, implement, and use practical measures 
in the context of inquiry cycles. These three challenges include designing practical measures specific 

to algebra engagement, collecting and analyzing practical measurement data, and interpreting the 
findings from their data to understand whether their teaching changes improved student engagement. 
BMTN teachers note that there may be many potential benefits of using common measures rather than 
designing their own. They hypothesize that it might streamline the process or build a stronger basis for 
collaboration with other teachers. In this sense, the diversity of measures used within the network might 
be contributing to the difficulty teachers had understanding the work of others and building upon it. 


Chapter 4: Challenges to Engagement in BMTN 


In addition to the challenges related to the integration of improvement science detailed in Chapter 3, 
BMTN teachers noted other challenges to network participation, and teachers who were new to the 
network identified different challenges than the teachers who were returning. One of the key issues that 
teachers had to address is their limited time in relation to the demands of the network. As is often the 
case with change efforts in education, finding the time to commit to the work is a significant challenge for 
teachers. BMTN participation requires both carved out time for in-person meetings as well as embedded 
time for inquiry execution and support structures. Interestingly, despite time being one of the most noted 
challenges, when an optional study group was offered that required additional time, the majority of 
BMTN teachers chose to join. This suggests they see great value in network activity. 


A number of teachers also noted that the work (both the inquiry and student-centered focus) was often in 
tension with aspects of their school context. Those with more rigid curricula and pacing found it difficult 
to inject student-centered activities while still meeting pacing expectations. Others found it challenging to 
fit in the data collection needed for inquiry given already very full class periods. 


Veteran teachers observed differences between network design in the previous year and found less 
momentum, support, and engagement this year as a result of changes made by network leaders to 
move to a more sustainable network model. New teachers were more likely to report typical feelings of 
uncertainty at the outset of engagement while they developed a sense of the work and process as well 
as challenges finding good math tasks to align with their change ideas. 


Overall, face-to-face meetings appear to serve an important function both for momentum and for 
participant satisfaction with the work. Although other network structures are important, reduction in 
face-to-face time over the course of the year was evident to teachers and it was missed. As the network 
scales, there may be a sense of a loss of “intimacy” or personalization, as several returning teachers 
noted, and attending to the needs of both new and veteran teachers may become more difficult. 
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Chapter 5: Teacher Perceptions of BMTN Impact on their Teaching 


Teachers in the Better Math Teaching Network are working toward the network's common aim: to 
increase the number of students in algebra who connect, justify, and solve with depth. Network leaders 
use their knowledge of mathematics and mathematics teaching to press network participants to consider 
not only if their students are more engaged in student-centered practices around connect, justify, and 
solve, but to also consider if their students are engaging with quality. In other words, are students doing 
the work (e.g., a student is justifying her answer) and are they doing the work with quality (e.g., her 
justification is of quality)? 


BMTN teachers agree that their engagement in the network has influenced their teaching in a variety 
of ways. We briefly note the general influence that BMTN teachers report the network has had on their 
teaching, and then look more carefully at three ways in which teachers perceive BMTN had an influence 
on their work: 


¢ Influence on instructional planning 
* Greater attention to task selection and adaptation of existing tasks. 
* More intentionality around task and activity ordering. 
* Greater focus on conceptual understanding rather than procedures only. 
* More intentional use of data for examining student learning. 


* Influence on student-centered instruction 

* Teacher reports of amount of instructional time spent using student- 
centered activities has increased from pre- to post-network involvement. 

* Greater change observed for teachers who came in reporting lower levels 
of student-centeredness. 

* Teaching practices that changed include those that leverage peers to learn 
from one another in pairs and small groups and thinking and reasoning 
about math. 

* Teachers changed their own behaviors, including talking less and refraining 
from directly answering some questions or showing relationships. 


* Integration 
* Testing their change ideas beyond a target class. 
* Sustaining use of change ideas beyond testing cycle. 


From these data we can conclude that teachers’ engagement in BMTN has influenced their work. 


Chapter 6: Teacher Perceptions of BMTN Impact on Student Engagement 


In Year 2 of the Better Math Teaching Network, the teacher participants had a combined 936 students 

in their target classes. These target math classes included a range of courses, but the majority were 
Algebra |. In this chapter, we report on what we learned about whether and in what ways teachers believe 
that their participation in BMTN has impacted student engagement in algebra. The chapter first explores 
how teachers conceptualize engagement and deep engagement in algebra for each of the DEAs: Connect, 
Justify, and Solve. These analyses show that in some pockets of the work, teachers are developing some 
shared conceptions of deep engagement while in other areas, especially in Solve, there is more variation. 
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This chapter also reports data on teacher perceptions of impact on student engagement, including: 


* 90% of teachers report moderate or substantial impact on engaging more students. 
Many specifically report progress in engaging less confident students. 


¢ Nearly all teachers reported moderate or substantial impact on engaging students with 
quality in the DEA they worked on, with more teachers reporting more intensive impacts 
for Connect and Justify than for Solve. 


* Teachers report that student work now consists of more opportunities to collaborate for 
math learning, increased expectations around writing and explaining rather than computation 
or rote learning, and that the onus is more on students to do the thinking and to really 
understand what they are doing. 


* Teachers report that beyond engagement, students’ reading and writing in relation to math 
and math texts/tasks has improved, and students are developing a more conceptual 
understanding of mathematics principles. 


The data presented in this chapter rely on teacher perceptions of impact on student engagement rather 
than more objective measures of engagement. These findings suggest that BMTN teachers find value 

in the network for increasing engagement of their students in algebra. This is an important indicator of 
network sustainability and a potential predictor of increased student learning. 


Chapter 7: Scaling the Network 


As the Better Math Teaching Network continues to grow, design decisions made by the network leaders 
evolve and shape participant experiences. In this chapter, we share insights about five key issues in 
scaling a NIC as they relate to the growth of BMTN: 


* Meeting the needs of teachers at various stages of participation. 
Our data suggest that new members feel the weight of learning improvement science 
most significantly in their first year, followed by figuring out how to contribute to 
and leverage the resources of the network. This suggests that network leaders need 
to support teachers new to the network on specific, and different, elements than 
returning members. 


Maintaining a sense of connectedness within the community. 

As network membership grows, some returning members experienced a loss of 
intimacy or connectedness. Network leaders in BMTN worked to mitigate this shift by 
tweaking the design of whole group meetings and other participation structures. 


Knowledge management. 

As the network grows, network leaders have to find ways to consolidate the learning 

of network members, capturing promising ideas and sharing them in a way that is 
accessible to all members and that can remain a “living artifact” that reflects the ongoing 
learning as it emerges over time. 
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* Negotiating the tension between quality of the experience and quantity of teachers in 
the network. 
As BMTN grew, direct support from network leaders was reduced from previous levels 
of intensity. In Year 3, network leaders are building new leadership roles for some 
returning members as a way to mitigate this shift in available resources. 


* Adapting the tools, routines, and roles for scaling. 
As the network grows, it is adapting the tools, routines, and roles to respond to 
participant needs in a more generative and sustainable way. Given what we have learned 
about variations in participant needs based on experience levels, shifts in the sense of 
connectedness, and the challenge of maintaining quality supports as the network grows, 
we offer a few considerations for ongoing design and implementation work in five 
categories: institutionalized mechanisms for transitioning new members, scaling access 
to expertise and rigor, scaling quality controls, differentiating work and roles, and 
managing knowledge. 


Chapter 8: Spread in Year 2 


One promising feature of an instructionally-focused NIC is that the field of education can potentially 
benefit from the NIC’s identification of promising instructional practices in order to accelerate 
improvement in other schools and classrooms. This chapter examines the formal and informal ways in 
which the Better Math Teaching Network worked to spread its impact in Year 2. Many BMTN teachers 
have engaged in informal spread to other math teachers individually and in department meetings at 
their schools. In these interactions, teachers tend to spread math-specific aspects of the work and ways 
to integrate improvement science into practice. In addition to informal spread, BMTN network leaders 
designed three formal mechanisms for the network to spread its impact, including: 


* Creating a public face of the network via Facebook, a website, and a monthly newsletter. 


¢ Encouraging BMTN teachers to present to the broader math education community via 
conference presentations and blogs. 


¢ Establishing a parallel network of school, district, and state math leaders (a leader network). 


There are a variety of ways in which ideas and new learning from NICs can spread beyond network 
members. Network leaders must intentionally design and manage these mechanisms, which in turn will 
focus both the ideas that are shared (e.g., improvement science strategies, classroom routines) and the 
audience for whom they are intended. An important job of network leaders is to consolidate learning, 
package it into deliverable components, and test out successful strategies for spread. 


Chapter 9: BMTN Spread: Building a District-Based Learning Community 


In Year 2, Better Math Teaching Network leaders spread the work of BMTN into a Rhode Island school 
district. They partnered with the high school’s math department chair to build a district-based learning 
community of middle school and high school math teachers that explored the implementation of 
improvement science using routines, tasks, and measurement tools tested and refined by BMTN 
teachers. This chapter tells the story of the first year of this learning community. 
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Conclusion 


The Better Math Teaching Network seeks to provide opportunities to increase student engagement in 
high school mathematics. Teachers in the network are highly engaged; they spend time outside of their 
typical teaching duties to attend quarterly network meetings and engage in quarterly inquiry cycles 
supported by virtual small group meetings. In addition, some BMTN teachers participate in optional 
study group meetings, support the network through leadership roles, and spread what they are learning 
beyond the network. Seeking to improve their own practice, BMTN teachers learn from each other and 
from network leaders. As the network moves into its third full year, it is poised to consolidate learning 
from several years of PDSA testing, build structures to spread that learning beyond network members 
into their schools, districts, and new districts, and inform the field on what it takes to grow and sustain 
an instructionally-focused networked improvement community. 
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Introduction 


The Better Math Teaching Network (BMTN) is looking to transform high school math instruction in New 
England and beyond. Through this network, researchers and practitioners are working together to 
make high school Algebra | classes more student centered. Launched in 2016 by researchers and expert 
practitioners at the American Institutes for Research (AIR), with support from the Nellie Mae Education 
Foundation, the network is grounded in the following five core principles: 


1. Teachers are central to change. Teachers shape 
students’ learning experiences and beliefs about math. 
It is possible to create classrooms that are more strongly 
student-centered—classrooms in which all students are 
actively and meaningfully engaged in learning math. 


This introductory narrative 


precedes every Better Math 
Teaching Network report, and 
was published in the BMTN 


2. Student-centered teaching is complex and almost Year One Developmental 
impossible to do in isolation. Teaching to maximize Evaluation Report, 2017, project 
student engagement and understanding is complex. staff: Jennifer Lin Russell, 

One way to deal with this complexity is for teachers to Jennifer Sherer, Jennifer Iriti, 
participate in structured, collaborative learning with and Courtney Long. 


other teachers and researchers. 


3. Teaching can be continuously improved. Teaching is a 
craft to continuously hone. Teachers use practices daily that lend themselves to 
ongoing, incremental improvement. Continuous improvement methods from industry 
and healthcare hold promise for education. 


4. Quick-cycle improvement methods provide opportunities to study and improve teaching. 
Many of the practices teachers want to improve can be studied with quick-cycle research 
and development methods. Teachers can test and refine strategies within and across 
lessons, realizing improvements every few weeks, rather than waiting until summer break. 


5. Research and practice should be seamlessly integrated. Too often, research and practice 
fail to inform each other. Our network includes researchers and practitioners working arm 
in arm to test and refine improvement strategies in real classroom settings. Mutual respect 
fuels our work. 


During the 2017-18 school year, 41 teachers from all six New England states participated in the second 
official year of the network's operation. Teachers were selected from a pool of volunteers that applied to 
be part of the initiative. Participating teachers work in urban, suburban, and rural contexts and teach at 
least one Algebra | course to ninth-grade students.' Teachers committed to work collaboratively to make 
their teaching more student-centered using the improvement science approach. The BMTN is organized 
as a networked improvement community, which is a structured network of researchers and practitioners 
working together to address a common problem of practice, in this case, student engagement in 
mathematics. 


1 Occasionally teaching assignments change, and in several cases, teachers returning to the network had no Algebra | sections in 2017-18. 
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Purpose: Student engagement in Algebra I 


Far too many American students are disengaged in learning mathematics (Boaler, 2002). Student 
engagement in Algebra | is a particularly pressing problem of educational practice due to the importance 
of the course in students’ academic trajectories. While Algebra | was once considered an advanced high 


school math class (Schiller & Hunt, 2011), more middle school students are currently enrolled in Algebra |. 


This shift has been driven by research that shows how early access to algebra is associated with students 
completing higher-level math courses in high school, which in turn predicts high school graduation and 
student postsecondary success (Stein, Kaufman, Sherman & Hillen, 2011). A 2006 study of high school 
students in Florida found that students who failed Algebra | were four times more likely to drop out than 
students who passed the course (Orihuela, 2006). Access to algebra content is an equity issue because 
students from lower-income families and those who are lower achieving tend to be tracked into lower- 
trajectory math courses, which amplifies achievement gaps in high school (Gamoran, Porter, Smithson & 
White, 1997; Stein, et al., 2011). 


Student-centered learning is at the core of the work of the Nellie Mae Education Foundation. In their 
framework for student-centered education, they present four tenants of student-centered approaches to 
learning: learning is personalized; learning is competency-based; learning takes place anytime, anywhere; 
and students exert ownership over their learning. The foundation theorizes that these tenets provide 
deeper learning for students which results in students building the knowledge, skills, and dispositions to 
succeed in college, career, and civic life (Putting Students at the Center: A Reference Guide). 


In 2014, the American Institutes of Research conducted a study, with support from the Nellie Mae 
Education Foundation, called An Up-Close Look at Student-Centered Math Teaching: A Study of Highly 
Regarded High School Teachers and Their Students. Through examination of case studies of highly regarded 
high school mathematics teachers, this study sought to break down the concept of student-centered 
learning in mathematics into key features. AIR researchers found that teachers enacted student-centered 
learning in different ways, but key common practices included the following: allowing for extensive 
student contribution, encouraging active student exploration, using problems that require students to 
think critically and communicate their thinking, and asking students to explain the “why” of their answers. 


Drawing on this study and prior research, leaders of the Better Math Teaching Network refined their 
stance on student-centered mathematics learning to focus on three principles for Deep Engagement in 
Algebra (DEAs): 


* Connect: Make connections among mathematical procedures, concepts, and application 
to real-world contexts, where appropriate. 


¢ Justify: Communicate and justify mathematical thinking as well as critique the reasoning 
of others. 


* Solve: Make sense of and solve challenging problems that extend beyond rote application 
of procedures. 


These three DEAs frame the improvement cycles that teachers in the network design and implement. 
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Strategy: Learning how to transform education through NICs 


The NIC concept 


The BMTN is organized as a network improvement community, or NIC for short. NICs bring together 
practitioners, educational leaders, and researchers in order to solve a pressing problem of practice 
(Bryk, Gomez, Grunow & LeMahieu, 2015; Hannan, Russell, Park & Takahashi, 2015; Russell, et al., 2017). 
In promoting the use of improvement science in networked communities, Tony Bryk and colleagues at 
the Carnegie Foundation for the Advancement of Teaching have promoted the NIC concept as a way 
for practitioners to learn how to improve education at scale by building an evidence base about both 
productive practices and knowledge of implementation processes to address persistent problems of 
practice and policy (Bryk, Yeager, Hausman, Muhich, Dolle, Grunow, LeMahieu, & Gomez, 2014). NICs 
are professional learning communities distinguished by four essential characteristics: 


1. They are focused on a well-specified common aim. 


2. They are guided by a deep understanding of the problem, the system that produces it, 
and a shared working theory of how to improve it. 


3. Their work is disciplined by the rigor of improvement research. 


4. They are coordinated to accelerate the development, testing, and refinement of 
interventions; their rapid diffusion out into the field; and their effective integration 
into varied educational contexts (Bryk et al., 2015; Russell, Bryk, Dolle, Gomez, 
LeMahieu, & Grunow, 2017). 


Jennifer Russell and colleagues (under review) argue that NICs aim to catalyze a community that can solve 
complex problems of practice. They refer to this as a scientific-professional learning community, a concept 
meant to emphasize a network's learning orientation, the collaboration among professionals that drives 
this learning, and a systematic approach to generating knowledge and practical improvement akin to the 
process utilized in scientific communities. Figure 1 represents the theory of NIC Development Framework 
proposed by Russell and colleagues. It details six domains of effort that are posited to be essential 
components of developing a network that operates as a scientific-professional learning community. 

This framework is a tool that guides the work of this developmental evaluation team, anchoring our 

tools and routines. 
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Figure 1: NIC Development Framework 


Addressed 


Improvement science (including quick overview of PDSA cycles) 


Improvement science is an applied science that has dramatically improved practice in a number of 
industries by helping practitioners learn their way into improvement. The approach has a long history in 
the manufacturing industry and subsequently the healthcare field. It provides a disciplined methodology 
for learning from practice to improve the systems and processes that shape work within organizations 
(Berwick, 2008; Deming, 2000; Gawande, 2007; Langley et al., 2009). More recently, education reformers 
and leaders have looked to improvement science as a way to accelerate large-scale improvement in 
schools and districts (Lewis, 2015). The teachers in BMTN use improvement science methods to improve 
their teaching, to refine their joint planning time (outside the classroom), improve the roles and routines 
students embrace, and make changes to structures or policies at the school or system level to allow for 
effective practices of student-centered formative assessment. 


Improvement science methods provide a disciplined approach for practitioners to learn how to improve 
work processes by introducing small changes (Bryk et al., 2015; Bryk, Gomez, & Grunow, 2010; Deming, 
2000; Langley et al., 2009; Lewis, 2015; Taylor et al., 2013). One central tool of the improvement science 
approach is the PDSA cycle. The PDSA cycle is an inquiry routine involving four steps: Plan-Do-Study- 

Act. The logic of the cycle is that practitioners learn how to improve their practice by planning a specific 
change tied to a working theory of improvement, testing the change, studying evidence to assess whether 
the change constituted an improvement, and deciding what action to take in light of what was learned. 
Identifying hypotheses, testing those hypotheses, and comparing results with one’s predictions generates 
new insights about how to improve practice. This rapid inquiry generates new learning; testing changes in 
varying sites of practice creates opportunities to examine variations in context that support or constrain 
practice. BMTN teachers use the PDSA cycle routine to test instructional changes and to see whether they 
improve student-centered formative assessment practices. 
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The power of networks 


Bryk and colleagues (2015) have theorized about how networked improvement communities can 

provide an organizational structure that helps practitioners learn to use improvement science to develop 
and test better work processes and learn from each other as tested innovations spread throughout a 
network. Across fields, networks have been looked to as a way to organize people to solve problems 

that require the integration of knowledge distributed across organizations and mobilize diverse social 
actors to engage in collective action (Kapucu, Hu & Khosa, 2014; Powell, Koput & Smith-Doerr, 1996). 
Networks intend to take advantage of a broad set of resources, as well as increase innovation, learning, 
and capacity building for problem-solving (Brass, Galaskiewicz, Greve & Tsai, 2004; Issett, Mergel, LeRoux, 
Mischen, & Rethemeyer, 2011; Kenis & Provan, 2009; Klijn, Edelenbos & Steijn, 2010; Weber & Khademian, 
2008). Networks can also facilitate the spread and implementation of promising solutions (Greenhalgh, 
Robert, Macfarlane, Bate & Kyriakidou, 2004; Valente, 1995). The Better Math Teaching Network aims to 
harness the problem-solving power of networks in pursuit of more student-centered teaching 

and learning. 


History of the Better Math Teaching Network 


The Nellie Mae Education Foundation is currently funding the Better Math Teaching Network, a 
networked improvement community (NIC) aimed at advancing the Foundation’s work to build an evidence 
base and the practical capacity to expand student-centered teaching and learning in New England. The 
American Institutes for Research (AIR) received funding in 2014 to launch and operate a NIC focused on 
student-centered learning in high school Algebra | as a potential solution to the problem of high rates of 
high school math students disengaged in mathematics learning. Network leaders from AIR spent roughly 
a year preparing to launch the network, working with the Carnegie Foundation for the Advancement of 
Teaching to learn about improvement science and the networked improvement community concept. 

In tandem, network leaders piloted some of their ideas for building a network with a group of nine 
teachers during the 2015-16 school year. During this time, the network also developed a working theory 
of improvement, and made the decision to focus on one component of that theory: making classroom 
instruction more engaging for students. 


The network launched officially in the 2016-17 school year with 23 teachers and continued in 2017-18 
with 41 teachers. In Year 2, the network held four face-to-face full-network meetings. During action 
periods between meetings, network leaders supported teachers’ efforts to identify and test changes in 
their practice that promoted deeper engagement in algebra content. During this second year of network 
operation, the participating teachers worked hard to learn and enact the improvement science approach. 
Their work was captured in a booklet that summarized their small tests of change and what they learned 
about student-centered learning in Algebra |. The network is currently in its third full year of operation 
and has expanded to work with 52 teachers representing all six New England states. 


BMTN is an important example of the use of the networked improvement community concept in 
education because it is trying to improve classroom instruction in Algebra |, a course that is an important 
milestone in students’ academic trajectories. While the networked improvement community concept is 
rapidly proliferating the education field, many NICs have not tried to address classroom instruction, in 
part due to the complexity of improving teaching. In this way BMTN is an important case for the field 
given the critical role classroom instruction plays in improving student learning outcomes. 
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Recognizing both the promise and challenges of networked improvement communities to support 
student-centered teaching and learning, Nellie Mae invested in an intensive developmental evaluation 
process exploring the network's development, outcomes, and dissemination of lessons learned. This 
developmental evaluation process aimed to provide timely and actionable information to network 
leaders and members, accelerating their capacity to meet their aim. Additionally, the developmental 
evaluation strives to produce useable knowledge for the education field and specifically other educators, 
policymakers and researchers utilizing the NIC model to organize for improvement and address high- 
leverage practical problems. 


This report outlines findings from the second year of this developmental evaluation of the Better Math 
Teaching Network. Our aim was to provide a descriptive and analytic portrait of the network's second year 
of operation. We drew on a range of data sources to examine this networked improvement community in 
action including: 


* observations of whole-group and small-group network meetings; 


* interviews with participating teachers conducted at multiple time points throughout 
the year (N=40 December; N=38 June); 


* teachers’ responses to a survey designed to measure teachers’ experiences with key 
features of the NIC concept (N=41 December; N=38 June); 


* analysis of teachers’ formal and informal connections to one another that are facilitated 
by the network; 


* documentation that teachers generate through their improvement cycles; 
* classroom observations and follow-up interviews with a small sample of teachers; 


* observations of meetings with BMTN affiliated educators (state, district, and school 
leaders; district-based learning community participants). 


Further information about our inquiry approach is available in Appendix A. 
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Organization of Year 2 Report 


We present our findings as chapters, which describe key themes that emerged from our 
analyses, focused as follows: 


Chapter 1: BMTN Routines and Tools 

Chapter 2: Influence of Routines on the Collaborative Work of BMTN Teachers 
Chapter 3: The Challenges of Improvement Science 

Chapter 4: Challenges to Engagement in BMTN 

Chapter 5: Teacher Perceptions of BMTN Impact on their Teaching 

Chapter 6: Teacher Perceptions of BMTN Impact on Student Engagement 
Chapter 7: Scaling the Network 

Chapter 8: Spread in Year 2 

Chapter 9: BMTN Spread: Building a District-Based Learning Community 


These chapters were written so they could be read either individually or as a set, depending on 
the interests of the reader. 


We believe our findings have implications for educators and education leaders who are interested in 
promoting student-centered mathematics teaching and learning. Additionally, educators and building 
or system leaders interested in the networked improvement community concept for organizing for 
practical improvement can see a portrait of how this network is organized and operated to support 
educator learning and teaching in more student-centered ways. Specifically, the first two chapters reveal 
how network leaders design for the work and how important these designed structures are to a healthy 
network. Chapters 3 and 4 examine challenges teachers face as they engage in the work. Chapters 5 
and 6 explore teacher perceptions of how their engagement in BMTN has influenced their own practice 
and the engagement of their students. In Chapter 7, we examine how the growth of BMTN has affected 
teacher experience in the network. We finish with a consideration of spread. In Chapter 8 we examine 
the different ways that the network is spreading the work and in Chapter 9 we present a case study of a 
learning community, which serves as one strategy for spread that the network leaders tried in Year 2. 
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Routines and tools are often used by leaders in organizations to structure work practices and to catalyze 
and sustain change. In the Better Math Teaching Network, network leaders designed and refined routines 
and tools to support both teachers’ use of improvement science and their engagement in the network. 
By intentionally and carefully designing these routines and tools, BMTN network leaders scaffolded 
teacher learning and supported teacher practice. In this chapter we examine three routines and three 
tools that BMTN network leaders designed and implemented. We consider each routine and tool's 
designed purpose. We then discuss the benefits and challenges teachers experienced as they engaged 

in each routine or used each tool, as reflected in our data. Finally, we note possible design implications 
for the field. 


Network meeting routine 


BMTN network leaders designed network meetings that convene all BMTN teachers 4-5 times each 
calendar year, with one convening in the summer and the remaining occurring during the school year. 
These in-person meetings provided opportunities to engage in new learning; opportunities to share ideas 
and tools; work time to plan, design, and collaborate; and social time for all participants to connect and 
build trust. 
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Benefits 


¢ BMTN teachers found these meetings to be the most useful element of the network in 
that they provided the opportunity for participants to meet face to face with each other 
and engage in a variety of ways (as listed above). 


* Teachers appreciated the time to work together with network leaders and other teachers 
on their Plan-Do-Study-Act (PDSA) work. Teachers saw this as important time to get started 
or maintain momentum on their individual work, and easily access a range of resources 
and expertise. 


* Teachers used network meetings to find new ideas, resources, tasks, and strategies for 
teaching student-centered Algebra | to ninth-graders from their BMTN colleagues. 


Challenges 


* Some teachers found the travel, time away from the classroom, and time away from 
their families challenging. 


* Some teachers returning to the network found redundancy in meeting content as new 
teachers were onboarded. 


* Interview data suggests that some returning teachers thought they were an underutilized 
resource in support of the learning of new members during these meetings. 


Virtual small group PDSA meeting routine 


Designed purpose 


Network leaders placed teachers into small groups for regular coaching and collaboration meetings based 
on the area of focus for their PDSA work (e.g., teachers trying out changes to engage more students in 
making mathematical connections were grouped together). Teachers in these groups met virtually at 
regular intervals four times throughout the year (in between in person network meetings) to share their 
PDSA work, explore challenges, and receive coaching support from peers and a network leader. Over 
time, in an effort to scale the work, network leaders released control of the meetings to participants by 
giving teachers a protocol to guide their collaborative time. 


Benefits 


* The majority of BMTN teachers found these meetings to be important or essential to 
their learning in Year 2. 


* Teachers appreciated the regular coaching, opportunities to check in, and the 
accountability that these meetings provided. 


* This routine allowed teachers to become more knowledgeable about what a few of 
their network colleagues were working on and to exchange meaningful advice and ideas. 


Challenges 


¢ While most teachers found the PDSA meetings useful, those who reported lower utility 
tended to report a combination of challenges, such as inconsistent participant attendance, 
unprepared group members, lack of a common focus among group members, participants 
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with very different school contexts, and/or groups that lacked the expertise to support 
their growth. 


¢ While many teachers thought the PDSA group meetings held without a network leader 
went smoothly, there was agreement that coach participation was critical in the early stages 
of the work, is likely critical throughout the year for groups with all new teachers, and added 
depth to the discussions which was valued by all groups. 


¢ Most teachers noted that it was at least slightly challenging to find time to meet virtually 
with their PDSA group, in part because teachers have different school day schedules, open 
periods, and responsibilities after the school day. 


Optional virtual study group routine 


Designed purpose 


Optional virtual study groups, focused on specific math content, were a new routine BMTN network 
leaders added in January of Year 2. As one hub leader reflected, “We added study groups to support 
teachers’ understanding of ‘depth.’ To really know what we mean by a deep connection, for example, 
they needed to have a deep understanding themselves.” The routine was designed to build content 
knowledge, pedagogical content knowledge, and to support disciplinary-focused collaboration. 
Network leaders selected texts, designed a protocol, and created groups of 3-4 teachers, with groups 
formed based on teachers’ choice of text and scheduling availability. Groups met to discuss one or two 
chapters of their book at a time; this meant that most groups met 6-7 times from January through May, 
approximately every two weeks. Twenty-four of the BMTN teachers (59 percent) chose to joina 

study group. 


Benefits 


* Both new and returning network members found the study group routine to be 
very valuable. 


* The greatest benefit noted in interviews was the value of working with other teachers. 
Reading about a math topic and discussing it with other math teachers provided a sense 
of belonging to a mathematics community, which is what many teachers who joined the 
network sought. 


* Other benefits that the study group teachers reported in interviews included building 
teacher knowledge and acquiring new resources. 


¢ Ninety percent of participating teachers indicated that they took something they learned 
in their study group and put it into practice [but not necessarily within their PDSA cycle] 
this year or would use it next year. 


Challenges 


* The biggest challenge teachers noted was difficulty finding time within their existing 
professional demands to meet with the group and the logistics of scheduling the meetings 
given the range of availabilities. 
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PDSA template tool 


Designed purpose 

Teachers in BMTN documented their inquiry cycles on a template that network leaders modified from the 
standard form used by the Carnegie Foundation for the Advancement of Teaching. The template frames 
the work in very specific ways: 


* It is organized by the four stages of the cycle (Plan-Do-Study-Act), signaling what 
specific information teachers should think about and fill in during each stage of their 
inquiry cycle. 

¢ It focuses teachers’ inquiry by having them fill in predictions and data for three 
questions: Will | implement the routine as planned? Will students engage in the routine? 
Will they engage with depth/quality? 

* Teachers archived their PDSA documentation in Google Drive. They embedded links to 


other relevant documents (e.g., measures used, student work, and compiled student data) 
so that network leaders and teachers have access to each other's work. 


Benefits 


¢ The template structured each step of the PDSA cycle for teachers, prompting data 
capture, analysis, and reflection. 


¢ Sixty-five percent of teachers thought that their PDSA documentation allowed them 
to accurately represent their work. These teachers thought the form served to structure 
their work and captured their practice so others could see it. 


Challenges 


* Thirty-five percent of teachers either did not think the PDSA documentation allowed 
them to accurately represent their work or they were not sure. Teachers in this 
category noted: 


* They often left things out of their documentation (e.g., learning was not well 
represented because of a rush to meet the deadline). 


* They were not able to capture their thinking/process in the template structure. 
* They were not able to document the dynamic work processes of the classroom. 
* They were not as reflective as they had hoped to be. 


* Teachers reported that the template was long and required attention to detail. Some 
teachers reported that completing the form was time intensive. 


* This created a tension for these teachers between the detailed work of 
documentation and time to reflect. 


* In some cases, teachers struggled to complete the form in real time, and 
consequently depended on memory to populate parts of the implementation 
and data reporting sections. 
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¢ While the template structured important elements of continuous improvement, some 
teachers struggled to fill in parts of the form where their knowledge of continuous 
improvement was still relatively nascent: 


* Making predictions and comparing results to those predictions 
* Aggregating data across trials 


* Articulating how their changes led (or did not lead) to improvements 


Student survey 


Designed purpose 


Network leaders designed a student survey tool to measure network progress toward BMTN’s aim. 
Administered two times each year,? students reported on how often they engage in different aspects 
of connect, justify, and solve (the three focal areas of deep engagement in algebra, as defined by 

the network). 


Benefits 


* Teachers reported this tool helps them stay more accountable to the aim and helps 
them intentionally design more activities that align with student-centered teaching and 
deep engagement in algebra. The tool is potentially a mechanism for encouraging 
teachers to implement student-centered practices beyond their PDSA cycles. 


Challenges 


* Teachers with inconsistent attendance, high student turnover, and/or different student 
populations for terms one and two wondered whether the trends noted in the data were 
accurately reflecting meaningful changes in student engagement. And so, overall, they 
wondered whether they could trust it as an indicator that their changes were leading 
to improvements. 


Change ideasummary template 


Designed purpose 


Network leaders designed two templates to support year-end reflection and to scaffold how BMTN 
teachers share their learning with the rest of the network. This was also a tool to help network leaders 
consolidate learning. 


¢ First, teachers populated a PowerPoint template by identifying the 
following elements: 
* Overview (DEA: Connect, justify, or solve; type of routine; unit/lesson timing) 


* Problem they were trying to solve 


* Change idea 


2 Some of the teachers who began new classes (and thus had new students) second semester gave the survey more frequently in order to 
capture pre/post data with each group of students. 
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* Key learnings 
* Final routine 
* Resources to support routine 
* Measures of engagement and deep engagement 
* Rubrics, results, student work and/or final thoughts, and next steps 
At the May network meeting, teachers presented their work to small groups of 


network members using this slide deck. 


¢ Second, teachers filled in a template that became part of a year-end book of change idea 
summaries that gets shared network-wide in the subsequent summer meeting with all 
new and returning teachers. The template included the same elements as the PowerPoint 
template, and the following additions: 


* Evidence of promise for the efficacy of the change idea 


* Description of the context in which they teach 


Benefits 


* Teachers found the change idea summary tools to be very helpful in supporting year-end 
reflection and learning across their individual PDSA cycles and templates. 


¢ Teachers appreciated learning from each other's presentations in May, and some used the 
change idea summary book as a way to learn from others. Roughly a third of the teachers 
used the book as a way to introduce themselves to the work. [See Figure 2 below. Note: 
Some teachers may have perused the change idea summary book but not used it at all or 
not in any of the ways listed. These teachers would have selected “none of these options.”] 
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Figure 2: Ways BMTN teachers report using the Change Idea Summary Book in Year 2 


None of these options 


It helped me get a sense 
for the work 


| edited a change idea, 
task, or measure from it 


| used a measure from it 


| used a task from it 


| used a change idea(s) from it 


It helped me connect with other 
teachers doing similar work 


10% 20% 
MM New Teachers @ Returners 


Challenges 


While the book served as a good introduction to the work, teachers rarely used it to identify a change 
idea or connect with others doing similar work, uses that might accelerate learning in the NIC. See 
Figure 2 above. 


Implications 


In BMTN, routines and tools helped network members navigate new learning and supported them in 
changing their teaching practice to improve student engagement. Routines and tools thus serve as a 
mechanism that network leaders can design and redesign as they support growth and change of individual 
members and the network as a whole. 


¢ Routines that were most effective involved regularity, collaboration, and opportunities 
for learning. 


* Balancing less frequent in-person and more frequent virtual meetings can support 
momentum, accountability, and engagement. 


¢ Simple tools work well for busy teachers, as does building in time for structured or 
guided reflection. 


* Finding ways to capture and consolidate learning from inquiry cycles is a persistent 
challenge for networks. 


In the next chapter, /nfluence of Routines on the Collaborative Work of BMTN Teachers, we focus on specific 
ways in which routines support the collaborative work in BMTN, and how critical the design of these 
routines is to the health of a network. 
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Chapter 2: Influence of Routines on the 
Collaborative Work of BMTN Teachers 


Network leaders designed and refined a series of tools and routines to support participant learning 
and engagement in the Better Math Teaching Network (see BMTN Routines and Tools). Three of these 
routines—often referred to as “participation structures” in networked improvement communities— 
organized the collaborative work of BMTN teachers: network meetings, PDSA meetings, and 
optional study group meetings. In this chapter, we examine the importance of these routines in 
building and sustaining collaboration, an element critical to accelerating learning in networked 
improvement communities. 


In addition to creating spaces for teacher collaboration, network leaders designed each of these BMTN 
routines to have a specific learning purpose as follows: 
¢ Network meetings: 


* Early in the year, the meetings focused on building common language and 
understanding of the aim (defining deep engagement in algebra) and the 
network's methodology (iterative improvement cycles, the PDSA). 


« Later in the year, the meetings focused on sharing learning and the 
consolidation of individual learning. 


e PDSA meetings: Teachers shared individual PDSA work and got coaching support 
from a network leader and other group members. 


¢ Study groups: Teachers built content knowledge and pedagogical content knowledge. 
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While nearly all BMTN teachers found these three participation structures valuable to their learning at 
some level, nearly three-quarters found the whole-group meetings to be essential. See Figure 3 below. 


Figure 3: Teacher perceptions of importance of participation structures 
for their learning? 


Whole-group 
network meetings 


Your small 
PDSA group 
Your study group 


0% 50% 


Mi Essential §§ Important = {J A little important Not important 


In addition, when asked about the adequacy of time for engaging in each structure, all teachers who 
participated in an optional study group thought the interaction was just right and nearly two-thirds of all 
BMTN teachers thought that their interaction with their PDSA group was just right. However, over half of 
BMTN teachers would have liked more whole-group network meetings. See Figure 4. 


Figure 4: Teacher perceptions about the amount of interaction in each structure 


The majority of teachers felt A little over half of teachers felt 

the amount of interaction with the amount of interaction with 

their small PDSA group was whole-group network meetings 
‘Just right’ was ‘Not enough’ 


3 In all, 24 out of 41 BMTN teachers chose to participate in a study group in Year 2. Percentages of study group responses are for N=24. 
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While BMTN teachers learned about the network's aim and methodology during network meetings, many 
teachers used network meetings as opportunities to connect with other math teachers and learn about 
new methods, tasks, and classroom activities. This network structure is also the only one that is face to 
face; the other meetings take place virtually. Some teachers also noted that network meetings filled a 
collaborative need that was missing from their home school contexts. 


Almost half of the BMTN teachers deemed PDSA groups as “essential.” For some, these groups offered 
support and accountability that helped them maintain momentum and stay on track. 


66 I love meeting with my PDSA small group. I felt like being a new 
member I think I needed a bit more regular, consistent support. 
Whenever I met with my PDSA small group, I left with a clear 
vision of next steps and where I needed to go with my testing. I 
would love more opportunities to do this.” 


I think for me the accountability piece to meet in the PDSA groups 
more often is a little bit better because I know I have to have this 
done by that next meeting. I think that might have helped keep me 
on track a little bit better.” 


Our small-group meetings were great because I would want to go 
into those meetings with everything current that I had.” 


Teachers who found the PDSA groups to be less important to their learning cited the following reasons: 
there was a lack of expertise in their group, members were not prepared for the meetings, and lack 

of participation (attendance) in the meetings. Thus, monitoring PDSA group health and intervening as 
warranted could be an important network leader function when such participation structures are used. 


Teachers who chose to be involved in study groups were overwhelmingly positive about the experience, 
even though the time demand was high: groups met 6-7 times, approximately every other week, 

and required additional work beyond the PDSA cycles that teachers were already conducting in their 
classrooms. Teachers reported the key benefits of the study groups to be working with other teachers, 
building teacher knowledge about specific math content, building student knowledge, and acquiring new 
resources. In the spring, as the frequency of other network participation structures decreased (due to a 
weather-related cancellation of the March network meeting and a decrease in frequency of spring PDSA 
meetings), study groups likely filled the collaborative void for many teachers. 


Overall, teachers emphasized the value of collaborating with other teachers through network structures. 
For example, one teacher described the value of these designed collaborative spaces saying: 


Better Math Teaching Network: Year 2 Developmental Report | Contents | 


31 


66 I think about how powerful the collaboration is. Even when we're 
not able to meet in person, but to have the smaller groups and 
the designated time to really focus on practice... Having like- 
minded people to talk to, people that are passionate about student 
learning, and focused on student-centered(ness) is really powerful.” 


In order to further explore teachers’ collaborative interactions in the network, we examined patterns 

of connections among participating teachers using social network analysis. Given that teachers place a 
high value on collaborating with one another through network structures, and a significant proportion 

of teachers had been in the network for multiple years, we wondered whether these connections 

might extend outside of formal participation structures. Analysis of the data suggests that this is not 

the case, at least on a large scale. While some deep connections have likely been established among 
subsets of teachers in BMTN, in general, collaboration among BMTN teachers happens primarily within 
the structured interactions and not informally outside of these designed participation spaces. Figure 

5 shows maps of interactions among BMTN teachers. The map on the left shows the connections that 
occur within designed collaborative spaces and the map on the right shows the connections that occur 
outside of formally designed spaces. Notably, 27 percent of teachers who responded to the survey do not 
interact with anyone else outside formal network structures. This suggests that the designed participation 
structures are critical to the network because they determine with whom and how intensely teachers 
collaborate with one another. 


Figure 5: BMTN teacher connections from formal structures (left) and BMTN teacher informal 
connections (right) 4 


4 Left map: Formal structure connections: PDSA small group, Study group, DEA time, and independent time (DEA/independent were 
combined, then only kept reciprocal ties). Right map: Outside of formal structure connections, reciprocal ties: email, phone, and in person. 
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Several teachers reported a desire to have additional interaction with other teachers but reflected on the 
difficulty of reaching out beyond scheduled meetings. For example, one teacher described the challenges, 


66 I tried to look through [Google] folders and reach out to some of 
the other teachers. But everybody's busy, and I didn’t get a whole 
lot back.” 


Another shared, 


66 Iwish I had taken advantage of my colleagues in the network 
more because | think that would've helped keep it on my radar. 
You always have a to-do list of things that you have to do and if 
it's not something you are talking to everybody about all the time, 
it tends to get pushed to the bottom of the list. I feel like I did that 
too much.” 


Some teachers noted their desire to work more with others but acknowledged that teachers are busy. 


6 I feel like I still have so much to learn, and I guess that’s probably 
been my biggest challenge. I’m the only teacher—some teachers 
have another person in the network who's also in their district, and 
so that close proximity to another teacher has allowed them to be 
able to work together and to bounce ideas off of each other. But I 
didn’t have that, so sometimes I felt that I really just wasn’t sure 
what I was doing was right, and that didn't feel good. It’s hard to 
feel like you don't know if youre doing the right thing... I think a 
part of it is probably just me reaching out. Like when I’m feeling 
like, “Oh, I could really use some help,” maybe trying to reach 
out and get that assistance on my own. But it’s hard, because 
everyone is so busy.” 


These teacher perceptions partially explain why designed collaborative spaces are so crucial for 
supporting sustained interactions within a network of educators. 
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Implications 


* Teachers lives are busy. Even when they are deeply committed to a network and 
the work involved, their collaboration is likely to be largely dependent on formally 
designed and implemented participation structures that organize meaningful work. 


¢ Although in-person meetings are resource intensive, teachers see face-to-face time 
as critical to the work. These face-to-face meetings build and maintain trust, 
motivation, and learning. 


* Given that participation structures largely shape the collaborative work that teachers 
do, network leaders must intentionally design them to achieve network goals. 
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‘estate 


Learning and integrating improvement science into practice is the biggest challenge teachers face as 
they engage in the Better Math Teaching Network. In end-of-year interviews, 53 percent of teachers who 
had been in the network more than one year and 78 percent of teachers new to the network identified 
an aspect of improvement science as one of the most significant challenges they encountered as they 
engaged in the work. Most often, they spoke about challenges with data and measurement. Survey data 
support these patterns, as is reflected in Figure 6 below. 
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Figure 6: BMTN teacher reports of challenges with using improvement science 
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Identifying a change idea 


Teachers also described identifying good change ideas and integrating improvement science work into 
their practice as teachers to be challenging. Some teachers mentioned not remembering to do the new 
routine or simply struggling to fit it into the flow of their classroom instruction. 


Typically, the first year of integrating improvement science into practice is difficult for teachers. As BMTN 
enters its third year, our interview data clearly indicates that teachers new to this work initially struggled, 
but after a year of doing the work, they were much more confident engaging in inquiry cycles. Almost half 
of teachers still indicated in interviews that some aspect of improvement science is challenging, even after 
their first year. 


There was a huge learning curve last year. And this year [the PDSA 
cycles] felt like part of my practice. This is just what we do.” 
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Another teacher expanded on this shift, 


It made it so hard for me that first year to come up with a change 
idea, collect some information, make a decision, and either 
embrace it, tweak it, or move on. This year was quite different. I feel 
like Iwas able to more swiftly move into and out of change ideas.” 


Once teachers became accustomed to engaging in inquiry cycles, some aspects of improvement science 
became less challenging for them. For example, once teachers engaged in a full PDSA cycle, they began to 
understand how to identify a change idea, go through the different steps of the cycle, and document their 
work. However, issues around data collection and measurement persisted into teachers’ second and third 
years of participation in BMTN, indicating that this challenge is harder to solve than teachers learning to 
integrate inquiry cycles into their practice over the course of their first year. 


Using practical measures in the context of disciplined inquiry cycles is a critical component of continuous 
improvement work (Bryk, Gomez, Grunow & LeMahieu, 2010; Yeager, et al., 2013). Practical measures 
are measures that operate at the level at which work is carried out. They are “practical” in that they can 
be collected, analyzed, and used within the daily work lives of practitioners. They are also “practical” 

in that they measure practice. Practical measures are used to learn whether the changes introduced 

are improvements. 


Challenges associated with designing, collecting, and using practical measures were frequently noted in 
interviews with BMTN teachers and supported by our analysis of their improvement cycle documentation. 
Below, we outline three challenges teachers experienced as they worked to design, implement, and use 
practical measures in the context of PDSA inquiry cycles. These three challenges were designing practical 
measures for algebra engagement, collecting and analyzing practical measure data, and interpreting the 
findings from their data. 


Challenge #1: Designing practical measures of 

algebra engagement 

Network leaders and teachers co-constructed definitions for each area of focus (connect, solve, and justify) 
and network leaders shared a “continuum” for justify and solve in the summer prior to the beginning of 


PDSA testing. Teachers then borrowed from these resources to design their own practical measures or 
worked with colleagues to design measures. See Figure 7 below. 
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Figure 7: Teacher responses to origins of their practical measures [multiple 
responses per teacher possible] 


Origins of practical measures for PDSA cycles 


Built it myself 

Co-constructed with BMTN collegue 

BMTN in-person meeting 

BMTN teacher in my virtual PDSA group 
BMTN teacher not in my small group 

BMTN Change Idea Summary Book for Year1 
BMTN virtual small group coaching meeting 
BMTN Hub leader 

Measures: Research 

BMTN teacher in my study group 

Other 

TRU math rubric 

Co-constructed with non-BMTN colleague 


Colleague from outside of the BMTN 


Network leaders encouraged teachers to think about engagement in two ways: are the students 
engaging in the work (e.g., are they doing the task?) and then, are the students deeply engaging in the 
work (e.g., are they doing the task with quality?). For this reason, most teachers built different measures 
for engagement and deep engagement. We examine these measures through the lens of our analysis of 
their change idea summaries—the documentation they crafted at the end of Year 2 to summarize their 
PDSA cycles. 


* Engagement: Teachers typically measured engagement by using a binary method: 75 percent 
of teachers who included an engagement measure used yes/no indicators for engagement 
(e.g., assignment/activity attempted) while the remaining 25 percent used some type of scale 
(e.g., 1-4 scale). 


* Deep engagement: The network defined deep engagement in three ways—solve, justify, and 
connect—and collectively built specific definitions for each of these “DEAs.” These definitions 
served as a framework for many of the practical measures that teachers designed to measure 
deep engagement. More than three-fourths of teachers (82 percent) who identified deep 
engagement measures used a rubric to measure deep engagement at different levels for varying 
criteria, while 15 percent used a teacher checklist to measure deep engagement as a collection 
of yes/no indicators. 
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Building measures was a frustrating experience for many BMTN teachers, due in part to their lack of 
training to design measurement tools. One teacher reflected on this challenge in an interview: 


66 Finding a way to collect data that wasn't skewed was a challenge. 
What did I consider quality justification? I think they talked deep 
justification. When I heard students speak [I would ask myself], 
when did I hear it and how many times did I hear it? Did you do 
deep justify once? Did you deep justify twice? It was really hard to 
capture that and tally that.” 


In this vignette, we see how one teacher grappled with measuring quality justifications. 


Vignette 1: Experienced, student-centered teacher 
digs deep into student thinking 


Joy teaches in a rural Vermont high school. Already an experienced and highly student-centered teacher, 
she joined the network to work with motivated teachers from other contexts so that she could continue 
to be exposed to new ways of thinking and working. Joy began her testing this year focused on the use of 
structured math talk to support students in justifying their reasoning around rules of exponents. 


Structured math talk includes partner work on a set of problems and then an exchange of the written 
partner work with another pair providing written feedback on their work. During this partner talk, Joy 
observes the student thinking evident within their discussions and written work. She uses what she 
observes to strategically highlight critical misunderstandings or questions with the whole class and have 
students help each other, 


/ write that down [student misconception] and | raise it in the whole group. | have a kid who says, 
“Arent we allowed to multiply exponents?” so we justified why you can multiply exponents. But 

it's at their questions and it is said in their words and they love to see it up there [on the board] 
because I'm not the one telling them. I'm not the one doing the work. They are doing the work. They 
are asking the questions. They are coming up with all the stuff and it’s really—that has changed my 
teaching so much because it’s all about them. 


Within this change idea, one of the key issues that Joy has grappled with is how to conceptualize and 
measure a quality justification, 


We have this rubric about what a quality justification is. But it hasn't really—the rubric has not been 
super helpful for me to decide if it’s a quality justification. What is a very good, easy measurable 
way to note what makes a good quality justification? | asked my math coach and she very easily 
said, a good quality justification shows that a student can do something in at least two different 
representations. So if they can graph it and solve it and show you the connection between those two, 
that right there is a good quality justification. If they can show you in the table of values and they 
can solve it, that’s a good quality justification. 
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Joy began working more intently on how students think through and use representations in their 
efforts to understand math and to justify their work, with an eye toward students’ use of two or more 
representations as evidence of justifying with quality. Joy used a subsequent PDSA cycle to think more 
deeply about how she can encourage flexible thinking and the use of multiple representations to 
strengthen students’ ability to justify their thinking with quality. The BMTN experience provided Joy with 
a structured and supported opportunity to delve deeply into her students’ thinking as well as how 
mathematics understanding can be identified and evaluated. 


Challenge #2: Collecting and analyzing practical measure data 


Once a teacher identifies or designs a practical measure, she has to integrate its use into her classroom 
and professional routines. Since engaging in PDSA cycles requires small changes and a quick assessment 
of those changes, the data collection, analysis, and reflection ideally takes place in a short window of time. 
Teachers face many challenges accomplishing this quick turnaround. Common interruptions to class time 
also disrupt the inquiry cycle (e.g., weather, assemblies, sickness, inconsistent attendance). As one 
teacher noted: 


66 When ordinary things happen within the school year it throws 
your schedule off it throws everything off, especially with the 
pressure I was putting on myself to get a certain point. Finally, I 
had to accept [that] it was okay to do two test rounds of the PDSA 
cycle and then I needed to move on.” 


Recording and reflecting on the data takes extra time teachers may not have on a regular basis. In some 
cases, teachers did not record, analyze, and reflect immediately; in these cases, they relied too much on 
memory to reflect on the efficacy of the test of their change idea. Since our minds filter observations, our 
present observations are affected by past observations (Langley et al, 2009). When teachers relied on 
memory, they may not have been reflecting on what occurred but what was most present. In this way, if 
teachers were not relying on actual data, but reflective constructions of what happened, they were not 
likely to benefit from the evidence. 


In other cases, when they got busy, teachers tended to let the documentation slip. As one teacher shared: 


Iwas so busy this spring I found it hard to keep up with the work 
for the first time. My change idea was around [justify], and I did 

a lot of that this year, but it was hard forme to document it all. I 
can see this year a little differently: one part is doing the work and 
implementing an idea in your classroom but another part of it is 
analyzing and documenting that work. That second piece became 
more challenging for me, particularly this spring.” 
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In some cases, the nature of the kind of data that teachers collected to assess students’ deep engagement 
in algebra posed a specific challenge. Given the nature of the practices they were trying to promote (e.g., 
justification), teachers sought to analyze student writing or student talk, which is naturally time intensive 
and difficult to integrate into the regular teaching rhythm. 


66 That was really overwhelming to me to listen to eight different 
conversations or nine different conversations. Even as I was 
moving about the room, I couldn’t spend enough time with one 
group—in my opinion—to get good data. So, the whole listening 
plece was a challenge for me, and that was a little disappointing 
because although I was successful with the students’ writing and I 
think that was helpful and there was growth there, we do so much 
more talking in math class than we do writing. I was really hoping 
to improve their discussion skills as opposed to their writing skills.” 


Due to challenges with collecting and analyzing data on talk, teachers often ended up with data that was 
not measuring what they had hoped to measure. For example, by narrowing their focus to writing about 
math because collecting student talk was too cumbersome, some teachers found themselves unable 

to capture how students were justifying their ideas through talk. This became a sticking point for many 
teachers focused on justify, especially teachers with English language learners. Typically, students are 
more able to talk about their ideas than write about them, particularly when they are building new 
understandings of complex ideas. 


Challenge #3: Interpreting and using data 


Once teachers had their data, and even after they had done some initial analyses of the data, some felt 
uncertain about how to interpret the results. For example, some teachers noted that they struggled with 
knowing how to see trends across PDSA trials. About 15 percent tracked some measure of engagement 
over time. About 25 percent tracked changes in a measure of deep engagement over time. But looking at 
trends in data over time was hard for other teachers. Some challenges included: 


¢ When the task differs so much as to influence student performance, teachers struggled with 
knowing how to compare two sets of data. 


e When teachers scored different sections of their rubric for different trials, they could not 
compare the scores across trials. For example, in Trial 1 a teacher might focus on only one 
aspect of justify, and then add other aspects to consider for subsequent lessons. 
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One teacher described her uncertainty in this way: 


6S I think my biggest struggle is that it took me close to the end [of the 
year] before I really understood that I was looking for a measure 
that said yes or no, the kids had deeper engagement. I've been 
collecting all these little pieces of things and then Id look at my 
data in the past and I was like, “So, what does that tell me?” I never 
put any end measure like, “If they do this, this, this, and this, this 
is what I'm calling deep engagement.’ I just kept collecting these 
separate little nuggets of information and then looking at it saying, 
‘Tdon't really know what this data is telling me.” 


A related challenge that emerged as teachers were trying to track changes in student engagement had 
to do with inconsistent student attendance. Several teachers noted it was hard to make judgments 
about changes due to inconsistent student attendance. This made it unclear whether variations in data 
from one trial to the next reflected a meaningful change or simply different student contributions to 
the data set. 


A key component of improvement science is “learning your way to improvement.” This, quite literally, 
means a teacher will use the data from her practical measures to determine if the small tests of change 
led to an improvement. In their change idea summaries, very few BMTN teachers reflected that their 
change had led to an improvement. It is possible that for some, the changes did lead to improvement 
and they simply did not record this on their final reflection. 


Lack of common practical measures may limit 
network learning 


BMTN teachers saw many potential benefits of using common measures. They noted how it might 
streamline the process or build a stronger basis for collaboration with other teachers. They did not yet 
know how to build them, indicating a possible area for network growth. Several teachers noted their 
desire for common measures in the network. In this sense, the diversity of measures used might be 
contributing to the difficulty teachers had understanding the work of others and building upon it. 


66 If we had a standardized rubric, would that make it less messy? 
Sometimes when we get together, we have five different rubrics, 
and they may not be evaluating the same justifying topics, the 
same solving topics, or the same connection topics. I wonder if we 
streamlined, if you would get similar data?” 
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It is possible that consistency of rubrics would help BMTN teachers understand each other's practice and 
use each other's change ideas more often. As one teacher reflected, 


66 I wonder if it would be a good idea for all of us to use the same 
rubric—lifwe had the] same rubrics for engagement and it was 
[a measure of] group engagement or discussion—just so that we 
dont have to flip back and forth between people’s rubrics to see, 
“Oh, what is engagement? What does that look like?” It can just 
be standard.” 


Identifying a set of truly practical measures that teachers can use to measure key aspects of 
deep engagement—solving, justifying, and connecting—may then support both teachers’ 
continuing use of improvement methods to learn how to better engage students and the depth 


of their collaboration with colleagues. 


Implications 


¢ Network leaders may want to consider ways to build common practical measures and support 
teacher groups as they make sense of the data they collect from common tools. Indeed, finding 
ways to support teachers to understand and interpret data from different measures may be 
an important element of NICs with members who are not trained in measurement. 


¢ In networks where members are building their own measures, network leaders may want 
to support teachers in understanding the difference between measurement and practical 
measurement. 
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Chapter 4: Challenges to Engagement in BMTN 


When engaging in an endeavor as ambitious as a networked improvement community, network members 
naturally face challenges. Primary challenges reported in the Better Math Teaching Network's second 
year generally fell into two categories: those related to the integration of improvement science into their 
practice and those related to their network participation. In addition, teachers who were new to the 
network identified different challenges than the teachers who were returning. In this chapter we focus 

on challenges as they relate to teachers’ network participation (i.e., network meetings and logistics of 
doing the work). We begin this chapter with a consideration of how time and context are challenges for 
BMTN teachers. Teachers new to the network had some unique challenges, which we then examine. 
Aside from the challenges of time and context, teachers returning to the network struggled with less 
access to coaching from network leaders, which was not surprising as leaders redistribute finite resources 
in a growing network. 


Time 


When interviewed at the end of Year 2, BMTN teachers often identified time constraints as the most 
significant challenge encountered in the work of this network. BMTN teachers struggled to find time to 
integrate the additional work into their practice. This includes time spent: 
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* in meetings (network meetings, PDSA group meetings, optional study group meetings); 
* traveling to network meetings; and 
* collecting, analyzing, and documenting inquiry cycles. 


As one teacher shared, 


A challenge was time—it's very time-consuming. It took a lot of 
my time.” 


Many BMTN teachers are involved in other professional activities in addition to their classroom 
responsibilities, and many have young children. These time challenges are reflected in interview 
data as well as survey data. See Figure 8 below. 


Figure 8: Challenges related to BMTN teachers identified in Spring 2018 


BMTN teacher challenges to network participation 


Hl Very challenging) Moderately challenging Slightly challenging [Not at all challenging 


Balancing BMTN work with your other 
professional responsibilities 


Time spent doing PDSAs 


Communication and clarity about 
expectations for your work in BMTN 


Finding time to meet with your virtual 
study group 


Time required for participation in network 
meetings & events 


Finding time to meet with your virtual 
PDSA group 


Logistics of participation in network meetings 
(e.g., travel, scheduling) 


Interpersonal challenges in your virtual 
small group (can be any group this year) 


Willingness of your virtual small group 
members to participate (can be any group 
this year) 


It is notable, however, that despite the time challenges, the majority of BMTN teachers persist in the 
network and opt to return the next year. In this second year of the network, half of the teachers chose 
to join study groups, which required allocating additional time to network activities. Therefore, despite 
their perceptions of time challenges, many teachers are finding ways to continue to actively engage in 
the network. 
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Contextual fit 


Another challenge BMTN teachers identified in interviews was the challenge of fitting this work into 
their teaching context, particularly with regard to curricular constraints. Some BMTN teachers identified 
challenges related to experimenting with practice changes given the constraints of mandated curricula, 
pacing guides, and district expectations for content coverage. One teacher explained, “It was challenging 
to fit it into our curriculum. We have a very regimented curriculum in terms of pace for the things we 
need to get through.” In the following vignette we show how one teacher found a way to successfully 
integrate student-centered engagement within significant curricular constraints. 


Vignette 2: Overcoming context constraints to incorporate more 
student-centered practices 


Matt's math classroom is governed by a department-wide curriculum and assessment timeline that 
often constrains what he feels he is able to do. Sticking to pacing for unit assessments makes efficiency 
a key factor in decisions about teaching approaches. Participation in the BMTN helped him transform 
his previously very teacher-centered test preparation routine into a student-centered formative 
assessment experience. 


! would say that the level of math talk in my classroom increased with this change idea versus the 
traditional way of implementing checkpoints. In the traditional sense the students would all work 
individually for a certain amount of time. Then | would go up to the board and go over the answers, 
show the work, and tell them this is worth this many points, this is worth this many points. And 
then at the end | would ask, “Do you have any questions?” And the students who were willing to ask 
questions would do so. And the students that werent willing remained silent and quiet. And they 
really weren't engaged with one another. 


He believed that students had mathematics expertise and could help each other if given the 
opportunity. He implemented a “tiered checkpoint” in which students worked on review problems 
independently, worked in pairs to discuss problems they did not understand, and then pairs joined 
into groups of four to further broaden access to their peers’ expertise. In each of these tiered 
moments, students were offered opportunities to reflect on their thinking and work with others to 
identify and attend to areas of confusion without the teacher providing the correct answer. He did 
this more student-centered activity in place of a more teacher-centered review he typically used 

to help students prepare for assessments. 


By doing the tiered checkpoint, at least it gave them the opportunity to talk with their partner pairs. | 
had students sit in two-by-twos and then in small groups. And [it] opened up a level of collaboration 
that | think, without the checkpoint, they wouldn’t get for a review for an assessment. 


Students engaged in this routine quite regularly through Matt's PDSA trials, and the routine became 
second nature for the students. In the end, Matt was able to carve out a niche of student-centeredness 
within his fairly constrained context. 
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When you're doing practice and reinforcement, | feel like sometimes that opportunity [student- 
centered practice] is lost because we’re aimed at just trying to get to the answers and making 
sure that the students have the solidified knowledge. But | wanted to definitely incorporate the 
engagement of communication, and justifying yourself, and critiquing others’ work. So, | think 
overall, those aspects of the change idea were useful in opening up those opportunities. 


After multiple rounds of PDSA trials in which Matt refined the routine, he incorporated it as part of 
his regular teaching practice when conducting reviews for assessments. Thus, within the fairly rigid 
pacing constraints in his department, Matt has provided his students with frequent student-centered 
experiences in order to engage more deeply with the mathematics concepts under study. 


dL 
Other teachers commented on how the new practices that they were experimenting with take more 
classroom time, and this presses them to adjust their content pacing. A few teachers commented on how 


scheduling constraints made the work challenging (e.g., class period is short, class only meets every other 
day, they get new students every trimester). 


&@ Our classes are only 45 minutes. It was hard to implement these 
new things into the curriculum that was already laid out for us, 
so that was certainly my largest hurdle. But once I got into a 
rhythm with the students and with myself in remembering all of 
the materials and data that I needed to collect, it became a very 
streamlined measure that I was able to finally manage. 


I think that the fact that we have trimesters plus a project 
month in between made it difficult. So, my pre-post test really 
is meaningless because it’s a different course two and a half 
trimesters ago and a whole different set of students. And those 
kinds of things that take some time to build into a culture just 
don’t happen at my school.” 


New teacher challenges 


In addition to challenges with time, context, and improvement science (discussed in the previous chapter), 
some teachers new to the network identified challenges in their first year that related to lack of certainty 
about the work and finding good math tasks. We examine each below. 


Lack of certainty about the work 


One-third of teachers new to the network in Year 2 identified a lack of certainty about what they were 
doing as they began the work as one of their most significant challenges. This is often the case as network 
members engage in a new, complex initiative. 
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66 It is confusing to Rnow what we're supposed to do for when... I’m 
an organization kind of guru. If I had received a sheet in July that 
stated, “On this date, this is due. On this date, this is due, and this is 
where you put it to actually turn it in,” think it would be better for 
me, to understand what's going on and then where I put it when 
I’m done. 


I think the biggest challenge for me was how to get started. 
Reflecting back on the year, I feel like September through 
December felt very confused and muddled and unsure of where I 
was going and what I was really testing. I think part of that was 
just being new to it. But getting started on this kind of work is kind 
of intimidating. That was one of the biggest challenges for me. 


At first, I didn’t really know where I was going with any of it. It was 
like the fear of the unknown.” 


Difficulty finding good math tasks 


When teachers new to the network were asked in interviews what their most significant challenge was, 
22 percent identified something related to math tasks. This included difficulty finding good tasks and 

a lack of time to search for good tasks. As one teacher noted, “It does take some time to be able to put 
together a good task for a PDSA. [Then it takes time] to really reflect and give the task its due diligence, 
to look through what you got from students based on the task you gave them.” 


Some teachers noted they needed training in how to identify and find high quality tasks. One 
teacher reflected, 


66 Thad a hard time understanding what kind of tasks I should be 
giving my students. So, there was a learning curve there....1 think 
we as a network should somehow share the tasks that we did in 
avariety of different content-specific areas. We might even show 
ones that don’t work and spend a little bit of time explaining 
why it doesn’t work and why something did work. I had heard a 
lot about low floor, high ceiling activities, but 've never had a lot 
of experience looking for them, finding them, using them in the 
classroom. I think maybe a little training on understanding how to 
go about finding those kinds of tasks would be very helpful.” 
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These teachers recognized that having good math tasks is a foundational resource for engaging students 
more deeply in algebra. In Year 3, the network is putting together a task library that will, in part, support 
this challenge. 


Implications 


¢ How might teachers in highly structured contexts select change ideas that will work within 
their constraints? 


* How might the experience of returning teachers be leveraged to support new teachers in 
overcoming their most pressing challenges related to engaging in the network? 


* How might the network support the use of good tasks, and support teachers in learning 
how to find and/or develop good tasks? 
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Chapter 5: Teacher Perceptions of BMTN 
Impact on their Teaching 


Teachers in the Better Math Teaching Network (BMTN) are working toward the network's common aim: 
to increase the number of high school students in algebra who connect, justify, and solve with depth. 
Network leaders use their knowledge of mathematics and mathematics teaching to press teachers 

to consider not only whether their students are more engaged in student-centered practices around 
connect, justify, and solve, but to also consider whether their students are engaging with quality. In other 
words, are students doing the work (e.g., a student is justifying her answer) and are they doing the work 
with quality (e.g., she gives a quality justification). 


BMTN teachers agree that their engagement in the network has influenced their teaching in a variety 
of ways. In this chapter, we briefly note the general influence that BMTN teachers report the network 
has had on their teaching. Next, we look more carefully at two ways in which teachers perceive BMTN 
had an influence on their practice: changes in instructional planning and changes in student-centered 
instruction. Then, we examine how those changes connect to the work that teachers were doing in 
BMTN—specifically their engagement in inquiry cycles. 


Overall influence on practice 


BMTN teachers report that as a result of their engagement in BMTN, they are more intentional in their 
teaching and more reflective about their practice, two key components of continuous improvement. 
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The work has helped me reflect on my practice to identify what 
Ido that supports student engagement. That has always been 
my goal, but I did not take the time to identify how to create my 
engaging classroom.” 


In addition, many of them report that they now think differently about how students learn. See Figure 
9. All but one teacher who took the survey said that the network has had some impact on their thinking 
about how students learn. 


Figure 9: Teacher assessments of extent to which network participation influenced their thinking 
about how students learn 


Amount of perceived impact of BMTN participation on 
teachers ‘thinking about how students learn’ 


No impact 
3% 
Substantial 
impact Minimal impact 
43% 8% 


Moderate 
impact 
46% 


6 We are leaning how to engage students in math and to think of 
a math class differently. I think that math has this reputation: 
your teacher has picked out all these different skills and tricks 
for you to learn throughout the year. Students don’t really know 
why they don't see the big picture, and I think that BMTN does a 
good job of helping students see that... they are all related and this 
is what you're building to, and this is how it connects with your 
previous learning.” 
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We don't know a lot about learning. We're also learning, what does 
it mean for students to justify their thinking? That is a very, very 
important question. I don’t think teachers generally think about 
that question enough.” 


We are asking, how are kids learning to solve? How are kids 
learning to think? How are they learning to justify their thinking so 
that others want to follow them down a path?... But don’t forget to 
get them to grapple with a problem and really try to dig in and see 
what they can come up. Do you recognize any patterns that you 
can use? How might you want to problem-solve this? Traditional 
math classes had nothing to do with teaching the kids to solve 

a problem on their own. It would be like, “This is how I did it. Now 
you do one that’s very similar.” We had kids that were very bored, 
and they werent learning anything. I think that getting people 

to shift their perspective on what's really important is something 
that we are learning and hopefully could spread around a little bit. 
Because, apparently, all that strict instruction wasn't needed to 
get kids to learn. I gave up a third of my time for that and it didn’t 
make any difference. So, I think we're on to something big there.” 


These teacher quotes illustrate how BMTN is spurring teachers to think more about what quality algebraic 
thinking looks like and how their own actions elicit that kind of understanding. 


Influence on instructional planning 


Teachers perceive their engagement in BMTN had an influence on their instructional planning. 
Instructional planning involves several elements: teachers must identify goals for student learning, select 
and adapt tasks and activities to engage students to reach those goals, and structure the class time to 
flow in meaningful ways as students engage. All of the BMTN teachers who took the survey indicated that 
their engagement in the network has had some impact on how they plan their lessons, with 37 percent 
noting a substantial impact. See Figure 10. 
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Figure 10: Teacher assessments of extent to which network participation influenced 
their planning 


Amount of impact of BMTN participation on how 
teachers plan their lessons 


No impact 


0, 
Substantial wide 


impact 


37% Minimal impact 


13% 


Moderate 
impact 
50% 


When we drill down to understand how the network has impacted planning, most teachers indicated that 
they used different tasks/activities, thought differently about how they would structure class, or both. 
See Figure 11. We also observe some cohort effects with more time in the network predicting a higher 
frequency of reported impact on how teachers structured class. Interestingly, the cohort effect does 

not hold for the selection of tasks, with fewer teachers from the earliest BMTN cohort reporting impact 
than their newer BMTN colleagues. This may be explained, in part, by the higher proportion of student- 
centered teachers in Cohort 1. 


Figure 11: Proportion of teachers reporting changed practice by cohort 
Change in practice as a result of BMTN participation 


HB Cohort 1 {MJ Cohort 2 Cohort 3 


| think differently about how 
| will structure class 


| use different tasks/activities 


l 
50% 
% of teachers selecting 
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Teachers also indicated the magnitude of impact of network participation on their task selection. Figure 
12 shows that more than half of teachers indicated a substantial impact with another nearly 40 percent 
indicating moderate impact. 


Figure 12: Proportion of teachers reporting various levels of impact on 
teaching practices 


Amount of impact BMTN has had on teacher practice 


Wi Substantial impact | Moderate impact || Minimal impact [JNo impact 


Selecting tasks 


BMTN teachers agree that the learning task they use matters in terms of engaging their students. In 
fact, this is one of the key learnings that BMTN teachers identified during the summer 2018 convening 
when they reflected on what the network has learned collectively. Teacher insights about the task seem 
to be centered on both the relevance of the task and the extent to which the task focuses on conceptual 
understanding rather than skill development. The following vignette reflects how one BMTN teacher 
thinks about task selection in her work. 


Vignette 3: Building the confidence to find and adapt 
high quality tasks 


Angelina teaches at a full-inclusion school in an urban school district. Her classes include both typically 

developing students as well as those with mild to severe disabilities. She works with a special education 
teacher and, together, they co-teach about 23 students in each of their classes. About 80 percent of her 
students qualify for free or reduced-price lunch and almost all are students of color. 


Angelina focused her BMTN work on justify. She wanted to support her students in making quality 
explanations for concepts in algebra, in part because her schools’ state assessments had shown 
weakness in constructed response questions where students needed to explain their thinking and 
justify in words. 


| got that idea from when we looked at our school-by-school data from last year. We noticed 
that many students did not score well on the constructed response on the state test. | think it 
was 30 percent of students actually got full credit on it, so we knew school-wide that this was an 
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instructional priority. We've been focused on writing across all content areas in my school for a 
while, and we were still unpacking state test data about how to improve this year. When we saw that 
it was constructed response, that’s when | came up with it’s not only the verbal justification, but it’s 
now what students actually put on paper. | knew that there had to be some kind of protocol or some 
way to collect data within our PDSA on the quality of what students were writing, so that’s where the 
justification in writing was born. 


As she started to think about how to help students justify their mathematical ideas, rather than rely on 
an algorithm, Angelina sought more rigorous and relevant tasks and adapted these tasks to incorporate 
more writing about mathematical thinking. 


The other reason that | wanted to work on writing was because | felt like when a kid is speaking, 
there’s a little bit of anxiety around it. Yes, the talk protocols help to increase talk in my classroom, 
but | felt like if | give students an opportunity to think independently and write on their own, that 
would increase the quality of what they produce. 


Angelina described using different tasks this year and using those tasks differently. She intentionally 
sought out a curriculum with tasks that provide meaningful context for students to justify their 
mathematical ideas in terms of relevant and accessible real-world applications. In addition, she 
adapted tasks to increase the opportunities for students to write about their thinking. She collaborated 
with two other network colleagues who work in her district to use an open-source math curriculum 
and work together to figure out how to adapt the curriculum for her purposes. 


It's a school-wide priority to focus in on writing, which is why my PDSA‘s trials are trying to 
incorporate low levels of writing. | want to build students up to using a really rich task where they 
have to do their calculations and then explain what that means. More traditional tasks would 
probably start with, “This is a system. This is how you solve the system,” without really providing 
students with a context, or having students work through that inquiry process. 


It's definitely a shift that I've made because of the network, and it's in the amount of writing that they 
do. Before, if the problem called for writing something or providing an argument, we would do that, 
but | wouldn'‘t plan for students to write any more than they needed to. But because of the network, 
I've been trying to incorporate more writing at low levels, just to build comfort with it. Because of the 
network, I've increased the amount of times | ask students to write. 


Angelina now feels more confident in making adaptations to existing high-quality tasks to deepen 
students’ opportunities to justify their thinking: 


Now, when | look at a lesson, | can figure it out. | can look on my own, without having to talk to a 
bunch of people. | can think about, “What would be a good supplemental question to insert here 
that would first clarify understanding?” | didn’t want the kids to be writing just to be writing. | wanted 
them to be writing about what we were doing, like something about the math, and that would help 
them clarify their thinking. In some way or shape, they’re thinking. 
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In addition to more deliberate selection and adaptation of the tasks used, BMTN teachers also reported 
that they are more intentional about how they think about structured routines in their classroom. 


66 I have been more intentional about my selection and sequencing 
of tasks. In my planning, I intentionally use routines from our 
BMTN work in order to engage students in persistence during 
private reasoning time, discussion, justification, and the critique 
of their own and others’ work. 


When planning, I had to make sure I was very intentional about 
the task I chose to be sure it led to being able to justify. I also had to 
be sure I would ask the right questions to probe student thinking.” 


In reflecting on whether BMTN had an impact on their planning, some teachers described a shift from 
a focus on skills to a focus on conceptual understanding. 


Iam choosing more robust tasks that elicit deeper connections 
and highlight important concepts rather than skills and have 
become more adept at talking about those important concepts 
and explicitly raising those connections with students.” 


66 I don't think I realized until I was taking the survey how much 
my planning has changed since joining the network. Iam more 
thorough and think more critically about the tasks used but also 
have almost eliminated note taking in all of my classes. I’m more 
conscientious about engaging all of my students in the task that 
I want to give out. I’m thinking more about the big picture or 
concept I want the students to take away.” 


Influence on student-centered practice 


Teachers also perceive their engagement in BMTN had an influence on their instructional student- 
centered practice. With the exception of a few teachers who were very student centered when they joined 
BMTN, each teacher in BMTN indicated that engagement in the network has made their teaching more 
student centered. Figure 13 shows the distribution of how teachers reported their student centeredness 
when they entered the network and now. The distribution from pre- to post-participation has both shifted 
to a higher reported frequency of student-centered practice and the overall variability has decreased, 
both indicators of systematic influence of network participation. 
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Figure 13: Distribution of BMTN teacher self-reported percentage of classroom time using 
student-centered practices before and during BMTN participation 


Self-reported student centeredness Self-reported student centeredness 
BEFORE joining BMTN in Spring 2018, after 1, 2, or 3 years 
of participation 
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When we ask teachers to explain why their practice had become more student centered, nearly three- 
quarters of teachers attributed the shift to their participation in BMTN. See Figure 14. 


Figure 14: Teacher attribution of student-centered practice change to BMTN 


Amount of change attributed to BMTN participation 


Not at all 
3% 
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To a great 26% 
extent 
71% 


Not surprisingly, teachers who entered the network with higher ratings of student centeredness reported 
less change, since they began with a lower growth potential on that dimension. See Figure 15. The data 
shows that those who entered the network with moderate to high levels of student centeredness did not 
report much change while those who entered the network fairly low on this dimension had medium to 
high levels of change in their student-centeredness. 
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Figure 15: Array of teacher student-centeredness before network participation 
by amount of report change 


Amount of change in student-centeredness 
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Many BMTN teachers reported an impact on student-centered math activities as a result of their 
engagement in the network. See Figure 16. We also measured the confidence teachers had in 
implementing these student-centered activities. Similar to some teacher reports on planning 
activities, some teachers report lower levels of impact on student-centered activities and high 
confidence, suggesting here, too, a possible ceiling effect. 


Figure 16: Teacher-reported impact of network participation on using 
student-centered practices 


BMTN impact on student-centered practices 
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The largest share of teachers reporting substantial impact centered on practices that leverage peers to 
learn from one another in pairs and small groups. When we consider both moderate and substantial 
levels of impact, the largest share of teachers reported the following student-centered practices: 

¢ Student opportunities for thinking and reasoning about math 

¢ Students learning from each other 


BMTN teachers who rated themselves low on student-centeredness when they entered the network 
reported greater network influence on their use of these student-centered practices than did colleagues 
who started off higher on student-centered practice. See Figure 17. 


Figure 17: Relationship between teacher-reported pre-network student-centeredness and impact 
of network on use of student-centered practices 
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The following vignette reflects one teacher's shift to student-centered practice. 


Vignette 4: Making ninth-grade math more student-centered 


Aymee entered the Better Math Teaching Network a veteran teacher. Engaged in leadership roles 
throughout her district and state, she consistently looked for ways to improve her practice. Years ago, 
she shifted her instruction to be more student centered and was confident in her practices. However, 
when she transitioned to teaching ninth-graders, she found herself teaching in more traditional ways. 
She explained: 


The big thing that caught me [about the BMTN] was the student centeredness. | had always been— 
past tense—had always been a teacher who facilitated a student-centered classroom. And then four 
years ago, my principal asked me to move from all my upper-level classes to take a freshman team. 
... | said okay. | got into the ninth grade, and all my student centeredness | threw out the window. It 
was a little traumatic because | found the level of maturity was not there. | couldn't do it with them 
because they wouldnt take it seriously. ... They weren't motivated. You put kids in groups, and they're 
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going to talk about math instead of what's happening last night around 9:00? It didn’t happen. So | 
threw it out. | was disappointed because the discovery wasn't there, and the investigations weren't 
there. They weren't owning their learning. 


She described her former practice: “| would model. | would get up and I'd be at the front of the room, and 
I'd be exhausted by the end of class. The responsibility was on me all the time.” 


Aymee's engagement in The Better Math Teaching Network changed that. Seeing other teachers engaging 
ninth-graders in student-centered practice inspired Aymee to implement a classroom routine she had 
done prior to teaching ninth-graders. She described the routine: 


A routine that is commonplace with every new task consists of: 


* 3-5 minutes of quiet individual thinking time, where students are 
asked to reason about the stimulus and formulate a plan of action 
or prepare clarifying questions 


* small group conversation about the stimulus: their first impressions, 
evidence of justification, components that are new, unfamiliar 
vocabulary, and 


* finally, a whole group discussion, which is as brief or extended as 
warranted by students. 


As students became comfortable with the routine, | allowed them to control the natural segues. 
In this way, what was once a rigid and stiff routine has merely become how we learn. 


Aymee built new tasks for her students to use as they engaged in this new routine. As a result of her 
BMTN work, she has a new approach to learning in her classroom: 


The old saying “Learn by Example” is exactly what happens in my classroom. Using the routine 
explained above, students consider an exemplary example of a task they will be asked to perform. 
They engage in a task that aligns very closely to the work in the example. Students work individually 
and with each other in randomly assigned groups. The work ethic in the room is palpable. Progress 
can be slow on day one; however, students will give feedback using various closing protocols and 
adjustments will be made to support learning and success if and when warranted. Often, a well- 
designed example and exercise set results in widespread success on the second exposure. 


Using inquiry cycles, Aymee has refined her routine and the way that she measures justification. 


My quest for the year was to encourage deep engagement in the justification. In the end, | learned 
that | needed to be explicit about what justification looks like in its varied representations. | used 
a rubric composed of indicators from the original BMTN Justify Deeply Rubric and the [state] 
Graduation Competencies Rubrics. Structure was once a primary criterion for quality justification. 
In the midst of my fourth PDSA, analysis and the effective use of multi-representations to support 
a conclusion became the primary criteria for quality justification. 
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Aymee has found success this year in a variety of different ways. In her words, 


* This new routine requires everybody to engage. It holds them 
accountable to themselves. They know that if they're not engaging 
and they're not doing the work, then they're going to be clueless 
the next day. 


It has engaged students differently because | ask them to make 
sense of a concept through reading and thinking and in conversation 
with each other. 


/ actually did a test that would be an Algebra II test. ... They're 
solving production, sales, revenue, cost-profit situation. My whole 
idea was justified that these kids going through an in-depth 
analysis making connections to multi-representations. And the 
work they perform is unbelievable, unbelievable. 


/ think the level of rigor [is there] because it's in application, it’s 
real world application, it's conceptual understanding and it is 
procedural, all in one. So, the rigor is there every single day. 
Every single day the expectations are high. 


Structuring class differently 


As teachers became more intentional about planning student-centered lessons, they structured their 
classrooms differently, changing the roles of both teachers and students. 


Changing what students do in class 


Many BMTN teachers report that they are increasing the frequency of activities in which students carry 
the “work” of the classroom. See Figure 18 below. 
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Figure 18: Teacher-reported change in classroom activities 


BMTN teacher report of student-related class- 
room activities 


My students work in partner or small groups 
more often 


Students talk more in class 


My students have more independent think time 


We have whole class discussions more often 


% of teachers selecting 


Changing what teachers do in class 


In making these changes to have students carry more of the load in class, BMTN teachers also changed 
what they themselves do during instruction. Some BMTN teachers noted that they think differently 
about their role in the classroom. For example, some reflected on how they are now more aware of the 
questions they are asking students. 


The hardest thing this year was coming up with the right kind of 
probing questions. I know what connection I want them to make, 
but how do I ask the question without making the connection for 
them? Right? And so that was really hard.” 


They are more likely to give students time to work rather than “swooping in and telling them how to solve 
their problems.” Figure 19 shows that up to half of BMTN teachers report changing their own behaviors 
during class time. 


Better Math Teaching Network: Year 2 Developmental Report | Contents | 62 


Figure 19: Teacher-reported change in classroom behaviors as a result of BATN 


Change in teacher behaviors as a result of 
BMTN participation 


| talk less in class 


| interact differently with students 37% 
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In the vignette below, one BMTN teacher reflects on how this shift worked for her in her first year in 

the network. She had always hung anchor charts on the walls of her classroom as a way to highlight for 
students the key learnings as they covered them. She changed this practice to be more student centered 
through her PDSA cycles focused on Connect. 


Vignette 5: Who is doing the work in the classroom? 
One teacher’s journey to help students make their 
own connections 


Stephanie focused her BMTN work on Connect. She wanted to support her students in making deep 
connections in algebra. 


Before my involvement with the network, I'd do a lot of guided notes with examples. Kids would 
diligently take the notes, and we would talk about them as we go. Hopefully the conversations would 
bring up connections, and | could explicitly show them the connection and how | thought about 

the connection. 


As she started to think about how to help students make their own connections rather than just listen to 
her make connections for them, Stephanie changed several things about her practice. She used different 
tasks, she engaged her students in class discussions, and she had students build anchor charts for 
themselves (and their peers in other classes) to use as references. 


* Tasks: Stephanie described using different tasks this year and using tasks differently. 
She intentionally identified tasks that provided the opportunity for students to make 
connections to that task (or from that task to other things they had learned), and then 
take the time to make sure students were making the connections. 


This year, | did a task, and it was much more student led. It always took longer. And then 
| had that prompt where | asked about connections instead of just the content. In the past, 
| may have said, “Here's a problem. Solve it using the shapes.” | did that this year, but | also 
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asked them, above and beyond that, “What's the connection between those shapes and a 
systems of equations problem that you would normally see?” 


e Discussion: 


[Last year my discussions were] more teacher led, posing an open-ended question and 
waiting for student responses. | never had students respond to each other. In my head 
| was going to try to make that happen, but | never really got there. And | still think I’m 
working towards it. 


This was a shift for her, as well as for her students: 


It's hard for them not to watch me to see if they're right or wrong. They’re so into, oh, no, 
| was right, or no, | was wrong. | think we’re still working on that. Everybody. Me included. 
... The questions | asked would have been open-ended questions around our content. 
This year it's definitely still open-ended around math content, but then making sure the 
connection is explicitly coming from students instead of me showing them the connection 
and saying, “Do you get it?” 


¢ Student-designed anchor charts with connections: 


In the past, | would have only had the content-driven [anchor charts]. | would not have had 
ones that were trying to explicitly show connections between that intro task to what we 
were doing throughout the unit. 


* Assessment: Toward the end of the year, Stephanie also started to think differently 
about assessment. 


I've changed this in my classroom and I'm collecting data for BMTN, but I'm not changing 
my assessments yet. It was too scary right now to change my big, end-of-the-unit 
assessment, so | wanted to see, if | captured these little blurbs [on their homework], what 
could | see from that? 


! would say that was a big change. Not only the task, not only how | ran the class, but also 
the things | was looking for as an end result of the task. 


Her involvement in the network allowed Stephanie to try out new practices in her classroom as she 
sought to support her students in making connections in algebra. 
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The process that facilitated change 


Engagement in the network has changed the ways in which teachers are planning and executing their 
lessons. It is also changing the ways that they think about their role, their students’ role, and how they can 
facilitate learning in their classrooms. In this section we explore how their changes are connected to their 
engagement in the core work of BMTN—the inquiry cycles. Specifically, we consider how inquiry cycles 
change the ways that teachers use data, how change ideas are integrated into practice, and how teachers 
sustain change ideas over time. 


PDSA cycles change how teachers use data 


The use of data within the instructional planning, implementation, and subsequent planning cycle can be 
a powerful mechanism for supporting student learning, as instructional decisions are tethered to actual, 
rather than assumed, understanding. All the teachers who took the survey report that the network has 
had some impact on how they use data to inform instruction. See Figure 20. 


Figure 20: Teacher-reported impact of BMTN on their use of data to inform instruction 


BMTN impact on teacher practice 


Using data to inform my 
instruction 


Wi Substantial impact [MJ Moderate impact Minimal impact [J No impact 


One teacher volunteered how integrating the use of data into her instructional decision-making was one 
of the most profound impacts of her participation in the network: 


66 Before I joined the network, I wouldn't say I was very data driven. 
As a school, we take interim assessments every quarter, and we 
use that data to create a plan, how to reteach concepts, and 
then reassess to see if students learn the material in a different 
way. I think that Iwas nowhere near the data-driven teacher 
that I thought I was since I joined the network because I’ve become 
so much more reliant on collecting data and figuring out how 
that data impacts student learning.... Now I’m using data 
to increase not only student achievement, but also student 
engagement, which has impacted my planning and my execution 
a great amount.” 
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Another teacher reflected on how she thinks differently about her use of data: 


6S I think that because I’m able to use a rubric to collect the data 
and to quantify or understand what qualities students are giving 
me, it helps me refine my data analysis skills. I think before, I was 
collecting data and making assumptions, and hypothesizing 
about why students struggled on some things. It’s just an 
assessment, right? Now, Im focusing on meaningful data, I’m 
focusing on the quality of the discussion, or the quality of a 
written response.” 


Change ideas become integrated into practice 


Half of the BMTN teachers reported that they tested their change ideas in more than one class in Year 
2. See Table 1. This is an early indicator that the work may become a part of routine practice for some 
teachers. The difference in reporting by cohort suggests that this is something that happens once 
teachers have been in the network for longer than a year. 


Table 1: Percent of teachers who test their change idea in more than one class 


Cohort 1 60% 40% 
Cohort 2 60% 40% 
Cohort 3 43% 57% 

Total 50% 50% 


Sustaining change ideas beyond testing cycles 


Through the course of their engagement in the network, BMTN teachers try out different change ideas. 
We have evidence that despite moving on to new ideas in subsequent cycles, many of the teachers 
returning to the network for one or more years are still using change ideas from previous cycles within 
their routine practice. According to interview data, all 15 teachers in their second or third year of network 
participation who we interviewed continued to use change ideas from previous years. Ninety-three 
percent of those teachers said they were using at least a small piece of their past change ideas in their 
classrooms this year. Some reflected on ways they had adapted and adopted what they did as a routine 
and incorporated it as they set class norms or classroom culture. 


& I would say I have implemented the basic routine that I tested last 
year, almost daily, in my class.” 
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66 I definitely still use that student-structured math talk as a way to 
get them [to justify] quite often. I would say probably if not once a 
week, a couple times every two weeks, as a structure that’s kind 
of become the norm. We then have students share out and make 
the anchor charts or make whatever it is that is going to become 
the public record. We keep those around the classroom, which was 
something that came out of the idea from last year.” 


Many returning teachers said they are using parts of past change ideas and are still fine-tuning them, 
although they are no longer collecting data on efficacy. A few noted that they are not using previously 
tested routines because they might be teaching a different class or using a different curriculum. Overall, 
evidence suggests that BMTN teachers are, to a large degree, using practices from previous change ideas 
because they observed positive effects on student engagement or learning. 


In this final vignette, we see how one urban teacher who has been in the network from the start uses the 
resources of the network—both people and material—to change her teaching practice. 


Vignette 6: The road to more student-centered practice: 
Putting the pieces together 


Anne works in a large urban high school that draws students from all over her city. Of the 900 students 
in her school, 63 percent qualify as economically disadvantaged, 53 percent are Latino/Latina, and 30 
percent are African-American. School starts at 7:30 a.m., and because many of her students take public 
transportation to school—some with commutes over 90 minutes—many students arrive to school 

late. She teaches on a ninth-grade team that loops with students through tenth grade, and her BMTN 
colleague—who is also on her team—teaches in the classroom next door to hers. 


In her three years as a network member, Anne has deeply changed her practice. The changes she has 
made have been largely influenced by her willingness to collaborate closely with colleagues in the 
network, her thoughtful use of the inquiry process, and her own willingness to take risks in her teaching. 
Anne's story is one that reflects the potential of NICs to influence the teaching practice of a teacher who 
is committed to continuous improvement. 


Fertile soil 


Anne was in her third year of teaching when she joined BMTN in 2015 as one of the network's pilot 
teachers. Her first two years of teaching were with Teach for America, after which she took a hiatus from 
teaching and completed a master’s degree in educational policy. She wanted to go back to teaching, and 
she was energized to make changes to her practice. 


/ was in this new space where | was ready to try new ideas and ready to refine my practice. This was 
a transition for me. | was consciously saying to myself, | want to continue teaching because | want to 
be better, and my students deserve that. 
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Setting the vision for student-centered teaching 


During her first year in the network, Anne's understanding of student-centered learning deepened. In 
the network, the pilot teachers watched videos of student-centered classrooms and examined discovery- 
based tasks. As Anne engaged in this work, she began to think about how her practice might become 
more student centered. Another teacher in the network helped her. Anne reflected, 


/ began asking myself, “What is student-centered teaching? What does that look like? What does that 
mean?” ... Hearing about [BMTN teacher's] experience and the way that she designs her class really 
influenced me in the beginning. 


A BMTN network leader (who was also her school-based math coach) pushed her to use more cognitively 
demanding tasks. 


It was the “new thing” and | wasn’t clear what that meant or how to use them in my classroom. But 
the network really expanded on that. | think seeing examples from [BMTN teacher] of how tasks are 
actually used in the classroom, from someone who had that much more experience than | did—that 
really helped. Understanding what a student-centered classroom looked like gave me a tangible goal 
to work toward. 


Anne noted that the first teacher whose work inspired her to think more carefully about student-centered 
practices taught in a very different context from hers. Anne sought out a second experienced teacher who 
worked in a context similar to hers to get more ideas and support. Anne's willingness to learn from others 
in the network helped her deepen her vision of student-centered teaching. 


Learning her way to deeper engagement: Connect, Justify, and Solve 


In her three years in the network, Anne engaged in PDSA cycles for each of the three DEAs (Connect, 
Justify, Solve). Her work in Connect, in her second year of the network, was her most productive. She 
reflected on how spending so much time on Connect allowed her to really refine what it meant to have 
students make connections. 


Previously | had always thought of making connections as making connections to a real-world 
scenario. | had not considered how to make deliberate and intentional connections to other math 
concepts. This expanded from, “How can | connect this concept or procedure to my students’ lives?” 
to “How can | connect what we're learning to my students’ lives and/or other mathematical ideas in 
order to build on their prior knowledge?’—whether it’s prior knowledge from that same year or from 
middle or elementary school. 


Her work in Justify was frustrating for her in the fall of her third year, mostly because she had planned her 
change idea around the students she had most recently taught (tenth-graders heading to eleventh grade 
whom she had looped with for two years) rather than the students she had (incoming ninth-graders). 
When her first PDSA cycle fell flat, she switched from Justify to Solve. 


| redid my process map and found | wanted to focus on “Solve” because my kids were really 
struggling with non-rote problems. | always heard them wondering, “I don’t even know what this 
is asking. What is this asking?” They just kept saying that over and over again. 


The work of some of her network colleagues helped her form her Solve change idea. 
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Hearing other people's ideas around solve helped me think about what it really means for students 
to engage deeply in problem solving. ... It boiled down to this idea that in life, you're always going to 
need to be able to use the information you're given and figure out how to use that information to 
solve whatever problem you are experiencing. This led me to focus on pushing my students 

to continually ask themselves, “What information are you given and how can you use it to solve 

the problem?” 


That other network colleagues were grappling with the same challenge validated for Anne that this was a 
good focus. Here again we see how Anne sought to learn from others as she defined a line of inquiry that 
best supported her students’ needs. 


Shifting to more student-centered practice 


The changes that Anne made happened over time, and in a variety of ways. First, she intentionally 
designed curriculum to facilitate connections. She rethought her role in the classroom on a daily basis 
and no longer filtered all of the information through her. “I started asking myself, ‘What am | doing that 
the kids could be doing?’ And pulling back on those things so that they are doing the heavy lifting, the 
thinking.” She used the “I do, we do, you do” method less frequently. 


She also changed the ways that she both supports students in her classroom and frames her lessons. 


| began asking myself, “What knowledge do students already have that they can build upon to learn 
this concept on their own? What new knowledge do | need to help them build?” Now | think about 
facilitating their learning instead of delivering knowledge. | try to give them instructions that will 
guide them to understanding or discovery. 


Finally, she structured her classroom space differently, clustering small groups of students together 
rather than putting them in rows that all faced her. 


Social aspects of the work supported Anne in making change 


Anne utilized the social aspects of BMTN to grow her practice. In addition to the two veteran teachers who 
helped her envision student-centered teaching in her first year, she worked closely with her school coach 
and one other BMTN colleague. She met daily with her BMTN school-based colleague to share work and 
co-create materials. In her second year in the network, her small PDSA group met in-person to collaborate 
outside of scheduled network meetings. 


She found that meeting with teachers in the designed participation structures (e.g., whole-group network 
meetings, small-group PDSA meetings) built accountability into her work. 


It’s exciting because | get to see all these great math teachers who have become my friends, and we 
get to talk about our ideas. ... When you're surrounded by good teaching and good teachers you 
naturally want to elevate your own practice. The environment fosters a space to think creatively and 
try new ideas. 


As aresult of her engagement in BMTN, Anne now reflects on her teaching in a completely new way. 
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This work has changed the way | evaluate how the day went. Whether it’s hard data, as in an 
assessment score, or looking at student work and grading it on a rubric or looking at how students 
were contributing to learning versus how | was—! reflect [through a] student-centered lens. And | 
think it actually allows for a lot more grace because it really is focused on the learning process. 


When she finds herself frustrated because the lesson is taking longer than she wants it to, she is able 
to turn that around and see the positive things that are happening: “The kids were working for the vast 
majority of the class period. They were working together. They were using something they learned 
previously to understand something slightly new today, and | didn't feel like | was the keeper 

of knowledge in class.” 


Sharing an anecdote from her day, her pride in her practice is clear: 


One student raised his hand and asked me a question. His group member—before | even 
answered—his group member said, “Why didn’t you ask me that? | could have told you that!” 
Reflecting on that moment | thought, okay, they know that they're empowered to do this learning 
and share this knowledge together. And so even though it took a lot longer than | wanted or planned, 
they were doing the work and they were doing the learning. 


Anne regularly sought out colleagues who could support her growth; she also was lucky enough to have 
BMTN colleagues in her building to work with for some of her years in the network. By engaging in change 
ideas that met her students’ needs and bringing a willingness to continuously improve to the work, Anne 
significantly changed her teaching practice to be more student centered and to more deeply engage her 
students in mathematics thinking and discourse. 


Implications 


* Teachers’ engagement in BMTN has influenced their practice. How can network leaders 
capitalize on the ways that teachers are changing their practice to articulate network 
learning, identify promising change ideas, and support the spread of promising changes? 


* The influence network participation has on a teacher varies based on their needs and 
expertise. How can network leaders design differentiated supports to meet varying 
teacher needs? 
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The work of the Better Math Teaching Network (BMTN) is driven by the following problem statement: too 
many high school students are not meaningfully engaged in learning algebra, limiting their opportunities 
to succeed in school and career. As a network leader shared in a network meeting early in Year 1, “The 
[Nellie Mae Education] Foundation was interested in disadvantaged, at-risk youth. We landed on Algebra 
| or the equivalent, in part because if you are a ninth-, tenth-, or eleventh-grader still in Algebra I, you 

are more likely to be at risk.” In an effort to solve this problem of practice, BMTN teachers focus on the 
aim statement: “to engage more students deeply in algebra by providing them opportunities to justify, 
connect, and solve with quality.” These three areas of focus (justify, connect, and solve) are referred to as 
DEAs—deep engagement in algebra. By engaging in continuous improvement, through using tools and 
routines of improvement science, the network supports BMTN teachers to target all students in their 
improvement work. 


The network's overarching design targets a diverse population of rural, urban, and suburban classrooms 
from across Maine, Vermont, New Hampshire, Massachusetts, Rhode Island, and Connecticut. In Year 

2, more urban schools in Massachusetts and Rhode Island were intentionally added, as those were 
underrepresented New England populations in the network's first year. 
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How many students might be impacted by BMTN? 


In Year 2 of BMTN, the teacher participants had a combined 936 students within their target classes.° 
These target math classes included a range of courses, but the majority were focused on Algebra I. 
See Table 2. 


Table 2: Math course identified by BMTN teachers in which they tested a change idea in Year 2 


Target Math course Number of teachers 


Math 1/Integrated Math 5) 
Algebra | 26 
Geometry 1 
Honors/Accelerated Algebra 2 2 
Algebra II 1 
Unknown 6 


Although many teachers did additional testing outside of their target classes and may have used learnings 
from their testing in other classes with other students that they taught, we focused our analyses and this 
chapter on teachers’ perceptions of changes in student engagement in their target classes. 


How are teachers conceptualizing student engagement? 


We analyzed the documentation teachers produced at the end of their inquiry cycles—their change 

idea summaries® —to extract BMTN teachers’ conceptualizations of engagement. In particular, we 

looked at the measures that they used to assess engagement within their PDSA cycles. Prompted by the 
distinction within the PDSA template, teachers generally offered their ideas for both engagement and deep 
engagement. Across all three DEAs, teachers generally conceptualized engagement along four dimensions, 
represented in Figure 21. 


Figure 21: Teacher conceptualizations of engagement based on change 
idea summary documents 


“ 
Teacher conceptualizations of engagement based on 
change idea summary documents 

Participation Attitude Effort Attention 
Attempting a task or Expressing openness Sticking to a task Actively listening 
activity (tried it) to a task even if it is difficult and/or focusing on 
a task 
os 


5 We refer to a target class as the class that a BMTN teacher tested out her change idea in a PDSA cycle. 
6 The change idea summary is the documentation teachers crafted at the end of Year 2 to summarize their PDSA cycles. 
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To assess these categories of engagement, teachers overwhelmingly sought evidence of students 
attempting a problem or assignment, actually completing an assignment (regardless of accuracy), and 
talking or contributing to discussion. Outside of these three common indicators, individual teachers also 
looked for student behaviors such as enthusiasm, listening, staying on task, amount of time engaged ina 
task, amount of work completed, and communication of ideas. 


In addition to engagement, teachers articulated how they conceptualized deep engagement and what they 
looked for as indicators of this concept. Deep engagement was conceptualized by teachers differently 

for each DEA as shown in Figure 22. We also included the number of teachers who submitted change 
summaries focused on each DEA. 


Figure 22: Teacher conceptualizations of deep engagement by DEA 
SS—<_S__— 00 SS SS Sa 


Teacher conceptualizations of deep engagement by DEA 


Connect Justify Solve 
(ED) EYAL) (N=8) 


* Quality Explaining Understand problem 
* Demonstrating Strategizing Reasoning & strategy 
understanding (@{o)| Fe} ole) e-lul ates Executing 


Analyzing, Revising 

synthesizing, & Correctness 

FTo)e) Nilarcaulavceya nate ia(eyal Restating 
(eo) nhitel=lale= 
Explaining 


Qe 


For the Connect and Justify DEAs, there was significant overlap in both conceptualization and indicators/ 
tools used to measure across teachers, suggesting that some shared language and measures have 
emerged among subgroups within these two DEAs. For example, 60 percent of teachers within Connect 
used a set of indicators that included referencing the big idea or mathematical concept, correctness, 
employing logical reasoning, articulating a common attribute or structure, and making a clear argument. 
Within Justify, common language and shared tools were even more prevalent, as 76 percent of teachers 
who focused on Justify looked for students’ use of mathematical concepts and relations, making a clear 
argument, and logical thinking as indicators of deep engagement with Justify. 


How teachers conceptualized deep engagement and what they looked for to measure it for the Solve 
DEA was much more varied. The analysis showed an occasional similar conceptualization; for example, 
one teacher might say “understanding the problem” and another would say “identifying important 
details” which are similar in meaning. However, there was no core set of common descriptors for deep 
engagement in Solve. Despite a few instances of these overlaps, there was little evidence of shared 
language across teachers working in this DEA, and the list of how teachers thought about Solve was 
broad, including: 
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* Discussion 

e Revising 

¢ Making sense of solution 

¢ Understanding the problem 

* Important details 

¢ Sensible approach 

e Attempting, monitoring, and changing 
e Reasoning 

* Strategy 


Confidence 

Behavior 

Improve understanding 

Relevance 

Correctness 

Supported 

Explaining 

Restating 

Reflection on strategy and approach 


Likewise, the indicators teachers working on Solve used to measure deep engagement were highly 
variable, with only one-quarter of teachers having any commonality. These common look-fors included 
students listing important information with reference, making sense of the task, and showing clear 
understanding in their own words. Outside of these, other indicators supplied by teachers included: 


* confidence in ability 
¢ follows group chat norms 


* gives clear explanation with precise language 


* shows conceptual understanding 


* solutions supported by representations 


* reflection on strategy and practice 


Overall, teachers indicated that it took a lot of time and some “false starts” to figure out definitions 
for deep engagement through their PDSA work as they grappled with how to measure in ways that 


had meaning: 


66 I think my biggest challenge is to figure out, “How do you have a 
measure that’s worth something, that tells you something?” You 
want to change your instruction based on data. Then, when you 


collect data, it’s not all good. So, that was a struggle for me. What 
does deep engagement really look like?” 


For some, after this struggle, there was a sense of satisfaction with settling on measures that embodied 


conceptualization of quality that matter: 


66’ think [my measurement now is] better because what I was flitting 
about with before, my numbers kept coming up super high, but 
as a teacher you say, ‘I don’t think they're working that hard and 
deeply engaged at trying to figure out why.” My kids were deeply 
engaged at trying to solve something, because they like to be right, 
but I don’t think they were working super hard and trying to figure 


out why things worked or why they were doing things. So, I do 
think I have something now that was working pretty well.” 
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Within DEAs, there was significant variation in how teachers are conceptualizing and measuring deep 
engagement. Within Connect and Justify there appeared to be some pockets of common language and 
indicators emerging, but this was not the case within Solve. This has implications for the network as they 
position themselves to arrive at and communicate promising solutions with a sufficient evidence base 
to warrant spread. Without a common conceptualization of what it looks like to So/ve with quality, it will 
be hard for teachers to build on each other's work. In addition, how teachers conceptualize engagement 
helps us to understand their perceptions of the impact of the network on student engagement, which is 
explored in the remainder of this chapter. 


In what ways do teachers perceive student engagement 
changing as a result of network participation? 


We conceptualize the potential impact teachers’ involvement in the network has on student engagement 
in four ways: engaging more students, engaging different students, engaging students differently within 
each DEA, and engaging for equity. We explore these potential impact areas by reporting teachers’ 
assessments of the extent to which their network participation has resulted in a change in each. 


Figure 23: Teacher assessment of impact of their 
work on engaging more students 


Engaging more and different students 


When asked on the survey about how much of an 
impact participation in the network had on engaging 
more students, 45 percent of BMTN teachers 
indicated a substantial impact, another 45 percent 
indicated moderate impact, and only 10 percent 
indicated either minimal or no impact. Thus, teachers 
were quite positive about the extent to which they 
have been able to engage more students within 
their target class. There were no significant cohort 
differences. Teachers in urban contexts reported 
slightly lower levels of impact compared with those 
in non-urban environments, but these were very 
modest differences. This data point did not show 
change over time from the fall 2017 to spring 

2018 surveys. 


Moderate to substantial impact 
on engaging more students 


In talking with teachers about whether and in what ways they believed that their change ideas resulted 
in more students being engaged, about 77 percent communicated a firm belief that they were engaging 
more students. Another 20 percent thought that they had observed at least some progress but were 
hesitant to make a judgment because it is hard to measure and/or they did not have concrete data from 
before participation to after participation. Only one interviewee did not think that her change ideas had 
resulted in engaging more students and this was because she reported already having extremely high 
levels of engagement. She viewed her PDSA work as an opportunity to experiment with powerful ways to 
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engage students differently rather than engage more students. Most commonly, teachers indicated that 
their change ideas did result in “more students doing an assignment” or “more students attempting all 
parts of assignments and investing more time.” 


Several teachers said that the PDSA process spurs them to carefully observe student engagement which, 
for some teachers led to new insights about individual students and shaped the support they provide to 
engage more students in more powerful ways. 


66 I think going through many iterations allowed me to watch kids. 
Because I wasn’t the one leading this, I could really see what they 
were doing and where kids were struggling, which gave me more 
opportunity to fix things so that more kids could feel like they were 


engaging.” 


Other teachers noted that the shift in student centeredness appears to have increased the frequency 
with which students share ideas during discussion, an important indicator of engagement for some 
BMTN teachers. 


66 Generally, when I give homework, we don't spend any time going 
over it.I don’t think it’s necessary. I don’t necessarily have a good 
valid reason for giving homework other than just to practice some 
more and for them to do some of the work outside of class time. 

I just take it and say, “Yep, you did a great job.” But this year I’m 
focused on the kids having conversations around it. They would 
actually come in and start talking about it before we had begun 
class. So, I would almost have to stop them and say, “Wait, we're 
going to have class time to do this so please hold off on your 
conversations.” But they really wanted to talk about the facts and 
they were motivated by them. A few students told me that they 
preferred that style of homework, that they were being asked to 
use Skills, but it wasn’t a direct question like they were kind of used 
to getting for homework.” 


Interviews suggested that teachers believed their change ideas were helping less confident students 
have opportunity and voice to engage in the work. Typically, teachers spoke of these students as having 
less confidence, in part, because they had lower levels of math skills or math facts as illustrated by these 
two teachers: 
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66 Even doing things like having a role was really nice because kids 
that are not as confident are never going to volunteer to be their 
table leader. But when I start to randomly assign these, they 
suddenly had that opportunity. I think sometimes kids just don’t 
do things because they have never done it before. Then you get 
in this cycle of “I guess I’m not that kid.” So, mixing things up and 
kind of forcing them to move around and work with other people, I 
think just really upped the engagement.” 


66 I get engagement from kids that don’t know their math facts right 
off the top of their head. I have more spirited debate on those days 
than I do normally. I also feel like kids refer back to those days 
more so than other days. It stays with them. I definitely feel like 
my kids that tend to get marginalized or don't talk when it’s skill 
things, because they feel like math should be a simple answer and 
it should be a quick simple answer and they don’t know their math 
facts. I get much more participation on my PDSA days than when 
Im doing more skill-based activities.” 


Engaging students differently 


Teachers chose a DEA and then tested change ideas intended to engage students meaningfully in algebra 
concepts. To understand whether teachers believed that their efforts are indeed engaging students 
differently, we asked teachers to indicate the extent to which they believed that their change ideas 

have affected student engagement in connecting, justifying, and solving with quality. Figure 24 presents 
teachers’ estimates of impact for the DEA on which they worked. 


Figure 24: Teacher assessments of student engagement impact by focal DEA 


Perceptions of impact on engaging students with 
quality aligned with teacher's DEA of focus 


i Substantial impact [J Moderate impact Minimal impact /§) No impact 


Engaging students in 
justify with quality 


Engaging students in 
connect with quality 


Engaging students in 
solve with quality 


Quality is defined as depth. 
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Interviews with teachers helped illuminate how they thought they were engaging students differently. 
Three key themes emerged: 


¢ Students are more likely to collaborate to support math learning 
¢ Student work now consists of writing and explaining rather than computation or rote learning 
* The onus is more on students to do the thinking and really understand what they are doing 


One of the most pervasive student engagement outcomes that teachers attributed to their change ideas 
was students becoming much more collaborative: 


They go from private reasoning time to not asking the teacher but 
really just pulling in group members and sharing their ideas and 
thoughts. I think that that was more powerful than I thought it 
would be. I felt that they would just be saying, “Well, what I wrote 
was—” But instead they were bouncing ideas off of each other, 
and a lot of their original plans adjusted quite a bit through their 
talking with each other. A lot of students who had questions got 
their questions answered. I think that that was really powerful, 

it was more focused on other students, and collaborating, and 
joining ideas, rather than relying on the teacher as the only 
resource. I think that was a difference that I noticed that was 
really rich.” 


6 I think my change idea was encouraging collaboration. In the past, 
I think that a lot of the practice that kids did in my class was more 
independent practice, kind of working almost in isolation. I had my 
desks in my room set up in rows and although they were allowed 
to talk, they kind of didn’t talk to each other because they just were 
in their own workspace. So, my change idea had my room set up a 
different way this year. I had groups of three and four, in the way 
that the desks were set up. There was just a lot more collaboration 
and alot more sharing of ideas and helping others and that was 
a big part of the idea and getting students to work together and to 
work differently.” 


Teachers described offering more challenging and meaningful tasks to engage students, increasing the 
extent to which students engage in writing and explaining mathematics rather than computation or 
rote learning, 
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The intention of the project was actually to kind of veer away 
from doing several rote problems and just give them one good 
problem. And I told them I don’t want you to spend hours on this. 
Spend 20 minutes and then move on with your life, get whatever 
you can down. Do some writing, ask some questions, even if you 
can’t answer, that’s okay. And one of the questions that I asked on 
the paper was how long did you spend on this? Because I wanted 
the kids to reflect that they were spending less time on it this 

way and then I would get kids who spent like 90 minutes on the 
assignments. So I had a lot of kids who really were engaged and 
wanted to figure them out and work them out themselves. I saw a 
big improvement in that throughout the trials that I did.” 


66 My students had an easier time talking about what they were 
seeing. But I saw more engagement in the actual written 
explanation itself You know, a lot of students just like to show you 
the math, and they don't really particularly enjoy explaining the 
math that theyre doing. And I saw more of that, that explanation 
piece, especially in the written form.” 


Teachers also described how the changes they have been making put the onus on students to do the 
thinking and really understand what they are doing; the students have a more active role and do more of 
the “heavy lifting”: 


I showed them three pieces of evidence, but I didn’t tell them 
whether they were right or wrong. And so, they needed to figure 
out whether it was right or wrong, and I think that engaged them 
differently, because they weren't just solving.” 


66 I think that they engaged them differently in the way that typically 
was justification, I think. It's usually, well, just explain your answer 
or just write about the math. And my change idea involved kind 
of building a justification. So like giving them the pieces of what 
would make a good justification... It made it more of a puzzle. It 
was just a kind of a different spin on how justification can look and 
how kids can get familiar with it before jumping all the way to [a 
full justification].” 
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66 I think that having those routines in place has changed students’ 
mindset about math and has made it more about looking 
for connections and patterns and ideas rather than it being 
about rules or procedures. Or when we are learning rules and 
procedures, they know that I’m going to always ask them like, 
“Why?’ or “How?” or you know, “Where does that come from?” 
You know, “What allows us to do that?” So I think I did a little bit 
of that in my practice before this, but I think that having these 
routines and change ideas in place has flipped it so that’s more 
of the focus of my classroom. And the content is certainly being 
learned and is important, but it’s being accessed more and 
constructed more by students because of these routines that 
are in place.” 


One teacher summarized that these different ways of engaging students shifted the classroom culture 
such that there is now space and structure for broad participation in mathematics discourse: 


6&6 Suddenly, when youre trying to draw in and engage everybody, 
those people who love to over-dominate don't have all this free 
space anymore because, suddenly, other kids start to participate 
and theyre throwing out their ideas. They didn’t really Rnow what 
to do with that. They're used to being able to be in a group, run the 
whole thing, do what they want. That threw them off” 


&& It really took me probably—I don’t now—seven times of running 
this protocol that I have before they really went, “Oh, never mind; 
I'll wait my turn,” without somebody else saying, “It’s not your turn 
yet.” I think that’s really good for everybody. Nobody wants to be 
working in the group where somebody's trying to run it all. You 
dont really get to good stuff until everybody has a chance to be 
heard and to throw some ideas out there.” 


Engaging for equity 

To honor Nellie Mae Education Foundation’s equity agenda, we explored whether there was evidence 
that teachers were engaging their historically marginalized students more. Overall, about three-quarters 
of all BMTN participants who teach those populations believed that participation has had moderate to 
significant effect on engaging historically marginalized students (including those with individual education 
plans, English language learners, students living in poverty, and students of color). 
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Since Nellie Mae is particularly interested in racial equity, the developmental evaluation sought to 
examine the work of BMTN through a lens that foregrounds race. Thus, we identified teachers who 
work with underrepresented minority populations and examined their perspectives on how the work is 
influencing their practice and their students. BMTN teachers who work in schools in which greater than 
50 percent of the students are non-white and/or are eligible for free or reduced-price lunch (FRPL) were 
identified as teachers in high underrepresented minority (URM) contexts. Figure 25 shows that teachers 
working in contexts with high proportions of URM students were slightly more positive about the impact 
of BMTN work on their students’ engagement than their BMTN colleagues who are not in such contexts. 


Figure 25: Teacher perceptions of impact on students by URM context 


Teachers with high URM populations report higher 
intensity levels of impact than teachers with lower 
URM populations 


Wi Substantial impact [§J Moderate impact | Minimal impact No impact 


non-URM 


In interviews, teachers shared their views of how their participation in BMTN affected students from 
historically marginalized populations, including English language learners, those with individualized 
education plans, students of color, and students growing up in poverty. The key themes that 
emerged include: 


* Change ideas that incorporate explaining and discussing appear to support English 
language learners and those who may be below grade level in math 


* Moving away from rote problems with right/wrong answers to complex, conceptual 
tasks that are implemented with conversation and analysis supports academically 
low performers to engage 


6 I feel like 'vue been able to engage more of my [English language 
learners] since then, expressing themselves or writing it down. 
And Ialso know a part of that is in the culture that I build in 
my classroom.” 
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66 I think it’s helping students. For example, my students who speak 
Spanish tend to not contribute as much. My change idea was to 
have them talk to each other. When they talk to each other they 
help them find their voices in terms of math, right? I always say 
there's something about hearing yourself talk, hearing the sound 
of your own voice and I think that happened a lot more. Most of 
my students are students of color, low-income students, and I 
would say my change idea did help them speak about math 
more than they used too.” 


66 I think a big piece of that is rigor and not dumbing down the math 
for students .... To me, the best strategy to engage marginalized 
learners is to give them tasks that they deserve. You know, that 
the “smart” kids get. I don’t Rnow how to say it sort of smoothly, 
but if youre starting from a strong curriculum or a strong 
database of tasks, you know, that are well liked, that have deep 
conceptual understanding and then you're working on increasing 
engagement I think the real benefit is to marginalized students in 
the classroom.” 


66 I think 100 percent lof students] are free and reduced lunch at my 
school. Ninety-eight percent are students of color. So, I would say 
all of the BMTN work has had an impact because of the students 
that I serve.I do think that it has let me have those students 
think more deeply about the bigger ideas of math—and I think it’s 
allowed me to become a better teacher for them, naming some of 
the skills that—those overarching ideas that are behind everything 
and helped me set a classroom where they're able to make the 
connections between things in a different way than before.” 


Despite these positive assessments of impact of BMTN participation on learning for URM students, a few 
teachers were hesitant to test their change ideas with classes serving many students from low-income or 
other marginalized groups that were struggling academically. 
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We have a lot of poverty here. Theyre not in this classroom. My 
students who have really grown up in poverty are in my Algebra I 
classroom. [That class is] geared towards the fourth grade. That's 
just where they are [academically]. It was a new class this year. 
[The school] made a cohort out of those groups, and I was a little 
afraid to do my testing there because when they gave them to me, 
their goal was just to keep them in school.” 


This teacher's experience suggests that as the network continues to develop, network leaders might need 
to both encourage teachers to work with their classes that serve marginalized learners who are struggling 
academically and provide teachers with the supports necessary to ensure that the testing is helping 

these students. 


To what extent is there evidence of network participation 
affecting student success? 


Although the network's aim is focused squarely on increasing engagement rather than directly on student 
learning, we asked teachers whether they had any evidence that their students’ learning had improved as 
a result of their PDSA work. Most of those who reported some impact on learning reported the following 
learning outcomes: 


¢ Students’ reading and writing in relation to math and math text/tasks has improved 


¢ Students are developing conceptual understanding of mathematics principles 


Teachers generally either did not have “hard” evidence of these learning impacts or suggested that they 
could probably compile the data from archived student work but did not have it readily available. Since 
the focus of the PDSA measurement is on engagement, only a few teachers had actually tracked any 
learning goals in relation to their PDSA. 


One teacher had an interesting observation about how she shifted time to more student-centered 
experiences and saw similar test scores among these students as she had for previous students, making 
her question the value of how she had spent that time previously: 
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6S My PDSA sometimes takes 45 minutes to an hour and I did it 
sometimes every 3 days. So, I find it very interesting that these 
students were able to get to the same end with all the regular 
curriculum kind of work and still miss that much instruction time. 
When youre thinking about direct instruction and me presenting 
stuff, you would think, “If I’m going to give up that much time on 
this problem-solving basis, that I wouldn't have still reached my 
end goal with my regular school curriculum.” So, there something 
in this that tells me that they were able to learn better for 
themselves, maybe. Where's the data on that? I don't now. But to 
take away that much time from what I used to do kind of tells me 
that maybe what I used to do wasn't very important. But I think 
when they come in now, especially if they know it’s going to be 
one of those days, theyre really excited. So, I think when you get 
to a higher-level engagement and then you switch gears and do 
something else, I think they're just picking it up quicker. Because 
their test scores and everything when they came in were just 
average. They were no different than the kids that I had last year.” 


Implications 


* The data presented in this chapter rely on teacher perceptions of impact on student 
engagement rather than more objective measures of engagement. These findings suggest 
that BMTN teachers find value in the network for increasing engagement of their students 
in algebra. This is an important indicator of network sustainability and a potential predictor 
of increased student learning. In addition, network leaders have collected data from students 
to understand their perceptions of classroom practices around student-centered instruction, 
which also suggest that student engagement is improving (AIR: The Better Math Teaching 
Network: Lessons Learned from Year 2 Report). 


* Data suggests lower levels of perceived impact for those working on the So/ve DEA compared 
with Connect and Justify. Indeed, the work within Solve appears to be less coherent across 
network members than in the other two DEAs. Further efforts around network design and 
next year’s research might further explore these differences. 


* Teachers working in high URM contexts found the work to have moderate to substantial 
impact on their students’ engagement, reflecting higher levels than their peers who do not 
work with similar student populations. 
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As the Better Math Teaching Network continues to grow, design decisions made by network leaders 
evolve and shape participant experiences. In this chapter, we share insights about four key issues in 


scaling a networked improvement community as they relate to the growth of BMTN: meeting the needs 
of teachers at various stages of participation; maintaining a sense of connectedness within the 
community; negotiating the tension between quality of the experience and quantity of teachers in 

the network; and adapting the tools, routines, and roles for scaling. 


Meeting the needs of teachers at various stages of 
network participation 


Membership within BMTN is dynamic as a handful of teachers who participate in the work for one or two 
years leave while the network intentionally grows its membership by recruiting new teachers each year to 
join. At any point in time, the network is home to teachers who are veteran to the network and those who 
are new recruits, posing a design challenge for network leaders as member needs and levels of expertise 
vary. If these groups are to have their different needs met to engage in the network meaningfully, the 
network must be able to differentiate support in order to both scale and sustain the network. 


As new members join a NIC, they are asked to learn on a variety of dimensions. This learning typically falls 
into three categories: 
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1. Social: What does it mean to engage in a community of professionals working towards 
a common aim? 


2. Improvement science: What does it mean to engage in inquiry cycles, and how do | 
integrate that into my current practice? 


3. Content: What is the new content that | am learning, and how do | integrate that into 
my current practice? 


Our data suggest that new members were challenged most significantly by learning improvement science 


in their first year, followed by figuring out how to contribute to and leverage the resources of the network. 


Familiarity and practice are important; as members gain experience in the network, they become more 
comfortable with participation routines and improvement science methods. One teacher reflected at the 
end of her second year, “I think it’s a very different experience your second year. You just know so much 
more about what you're doing.” Another teacher shared a similar perspective, “I would say one of the 
biggest differences [between my first year and this year] is just my feeling of comfort. | felt very at ease 
knowing the system and what was expected of me.” 


This shift in comfort and confidence is particularly true in the area of improvement science. Since 
improvement science is a new methodology for almost all of the teachers who joined BMTN, learning 
this approach and integrating it into practice posed a challenge for nearly everyone. As BMTN enters its 
third year, our data clearly indicates that teachers new to this work initially struggled, but after a year 

of doing the work, they were much more confident engaging in inquiry cycles. “There was a huge learning 
curve last year. And this year [the PDSA cycles] felt like part of my practice. This is just what we do.” 
Another teacher expanded on this shift, 


6S It made it so hard for me that first year to come up with a 
change idea, collect some information, make a decision, and 
either embrace it, tweak it, or move on. This year [second year 
of participation] was quite different. I feel like Iwas able to more 
swiftly move into and out of change ideas.” 


This relationship between experience level in the network and confidence using the PDSA routine is also 
reflected in the survey data below, with the newest members (Cohort 3) reporting the lowest confidence 
levels. See Figure 26. 
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Figure 26: Comfort with PDSA routine by cohort in Spring 2018 
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Teachers newer to the network and those who have participated for one or more years shared several 
challenges but also reported some that are unique. These differences are particularly important to attend 
to as the network scales; without supporting participant needs the network may not be able to sustain 
membership. See Figure 27. 


Figure 27: New and returning teacher key challenges 
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New teachers reported more challenges overall, and more of their challenges focused on transitioning 
into the work. Returning teachers mentioned more challenges having to do with the substance of the 
work, including lack of fit with what they want to work on and their need to interact with others who have 
expertise that could help them improve their practice. 


Interestingly, both new and returning teachers reported the ongoing challenge of measurement. This 
issue is highlighted in The Challenges of Improvement Science chapter. In this chapter, we consider the 
scaling implications of this measurement challenge. While teachers in the first year of participation build 
comfort in the PDSA inquiry cycle approach to practice improvement, teachers in their second and third 
years of participation tended to struggle with how to identify, collect, and analyze data to know whether 
changes are an improvement. This is an area in which BMTN teachers were least confident. Figure 28 
below shows low intensity of agreement across all cohorts, which indicates that the measurement 
challenges are present from the start and persist. 


Figure 28: Teacher confidence in their PDSA measures by cohort 
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These differences are important to consider as the network scales and remains responsive to member 
needs. It is important for network leaders to consider questions like: How will the network support 
new members’ learning about the PDSA cycles in a scaled model? How will teachers’ use of practical 
measurement be supported as growth in membership is realized? 
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Maintaining a sense of connectedness 


As the network grows its membership, there is a potential for the sense of connectedness within the 
community to be reduced. In particular, the number of people at the face-to-face meetings can reduce 
the sense of intimacy as this teacher observed: 


6 I think the meeting felt a little different [this year] because there 
were just so darn many people. I think that I tried to not say as 
much during the meetings because everybody wants to comment 
on every little thing. It can be redundant, and it also gets us bogged 
down and you can't complete an agenda. In trying to be respectful 
of the agenda, I felt like I wasn’t participating as much and that 
kind of left me thinking, “Well, did I stay too quiet?” You get into 
that dilemma of you don’t want to monopolize, so that kind of left 
me a little confused on what I should be doing there.” 


Some teachers reported changing their participation and engagement practices, partly in response to 
the size and structure of the meetings. Whole-group portions of the meetings became more challenging 
for participants to feel both active and productive. Participants also noted that the increased size made it 
harder to know what is being tested across the network. 


I think the bigger we get, it’s harder to talk to as many people who 
are doing different testing. So, for that stuff, I'll hear somebody talk 
about something that they are trying, and I'll go look in their PDSA 
[folder] so I can get ideas for things that are working or not. But it’s 
harder to get all of that in a sitting when there are just so many 
people there.” 


6 As the network is getting bigger, the NIC is almost too big for it to 
feel like a community. I can't keep track of what everyone's doing 
because there are too many people. When it was smaller, I knew 
what everyone was doing in their classrooms, and so it was easier 
to learn their story, hear how things were going, and be able to 
keep track of it along the way.” 


As the network scales, a key challenge is for participants to be able to see the whole of the work of the 
network in a concise and digestible format. In this way, scaling a network puts greater demands on 
network leaders to consolidate what network members are learning and share that learning with the 
full network. 
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Survey data from the first two BMTN cohorts collected at two points in time (Fall 2017 and Spring 2018) 
suggest a small but meaningful decline in perceptions of opportunities to share with network leaders, 
with other teachers, and to learn about what other teachers are doing. See Figure 29. 


Figure 29: Change in teachers’ perceived opportunities to connect and share learning 
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These trends may have been influenced by the fact that the March network meeting was cancelled due 
to weather, suggesting that the face-to-face meetings are critical for maintaining a sense of connection 
in the learning. 


Knowledge management 
Knowledge management is an important function that NIC leaders carry out. In order to tap into the 
potential of the network to accelerate learning, network leaders must find ways to: 

* harvest promising change ideas, 

* assess whether the ideas are fruitful, and 


¢ make the ideas available to others in a way 
that is easily accessible in order to test in 
varying contexts. 
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This is challenging and complex work, particularly in an instructionally focused NIC where teachers’ 
change ideas can be context-dependent, tied to specific curricula, or narrowly focused in other ways. 

In addition, this work happens once NICs have been launched, trust has been built, and tools and 
routines are in place to support knowledge management. At the end of Year 2, BMTN is poised to begin 
this work. While network leaders have elevated some change ideas and pushed them out beyond the 
network (see BMTN Spread: Building a District-Based Learning Community chapter), they have not yet built 
the mechanisms for identifying or spreading ideas internally within the network, nor have they explicitly 
determined when and how to spread ideas (e.g., by defining when a change idea is worthy of testing 

by others). 


As a result, when asked what the network is learning collectively, 79 percent of teachers interviewed were 
unsure. While they are highly excited by the work, and they can talk about what they themselves are 
learning, BMTN teachers are less sure about what the network is learning as a whole. Of those who are 
unsure, several of them believed that it would happen or that perhaps it was happening, but they did not 
yet know about it. In general, teachers focused their responses on the network's aim, defining the work of 
the network, and what teachers are learning. 


Developing tools and routines for managing knowledge might mitigate these teacher concerns about not 
knowing what others in the network are testing as well as supporting ways in which teachers can identify 
the learning of other teachers that might be of use to their own learning. 


Ensuring quality support as the network grows 


As the network grows, direct support from network leaders must naturally shift, and, in effect, be 
reduced from previous levels of intensity (unless, of course, the leadership team grew in tandem with 
network membership). In Year 2, network leaders facilitated PDSA small groups in the beginning of 
the year. Network leaders did not attend the final meeting of the year; instead, teachers ran their own 
small PDSA groups with a leader-designed protocol. In Year 1, network leaders engaged in iterative 
revision work for PDSA change summaries but did not continue that practice into Year 2. Returning 
teachers felt these shifts: 


&6 Part of the challenge was that I felt like we were a little bit more 
on our own this year. Because there were more people, we didn’t 
have our small PDSA group meetings like we did previously. We 
would meet and talk and stuff like that, but we didn't have network 
leaders as part of [all of] those meetings, so that was a little 
difficult because there were times where we had questions, or we 
were just like, “Is this what we're looking for?” 


In particular, the role of the network leader as both an outsider and an expert during PDSA meetings 
was missed by many returners: 
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G [Network leaders] always give us a great perspective. It was still 
helpful [to have the meeting without them], but I would have really 
liked to have had them [in that last PDSA meeting]. Sometimes, 
ifwere all having the same question, it takes somebody who's 
not deep in the work to remind you of what the goal was. I think 
that’s the part that [the network leaders] always play for me. [The 
network leaders] are always like, “Yes, but we're talking about deep 
engagement here’—-somebody to keep you on track because it’s 
very easy to get off-track.” 


As the network scales, network leaders cannot possibly continue to provide the same support to all 
participants. In year-end interviews, teachers indicated how they thought the first meetings went without 
a network leader. About half thought that the meetings went “well,” 36 percent thought they went “ok,” 
and about 14 percent thought they went poorly. We followed up by asking if teachers thought that 
running PDSA groups without network leadership was sustainable. About 28 percent thought it was a 
sustainable approach, 58 percent thought they could possibly be sustainable, and 14 percent thought 
they were not sustainable without network leader facilitation. Some teachers offered the following 
suggestions for sustaining these meetings: 


¢ Early PDSA group meetings need expert facilitation to establish routines and culture. 
* Groups should never be comprised entirely of new teachers. 


* Someone else, perhaps a trained veteran teacher, could fill the facilitation role 
(which the network is currently implementing in Year 3). 


6 If it's going to be sustainable, then it cant just be on the network 
leaders. They have to look to others within the network to help. 
You know, other [network veterans]. People who can help with 
those meetings, who can be coaches, help with the whole-network 
planning ideas. Especially if there are going to be 16 more teachers 
next year It’s just pretty hard. And not sustainable.” 


¢ Network leaders should provide clear ways for teachers to access their expertise and a 
structure for this access so that teachers do not perceive that they are asking for something 
“extra” from network leaders. 


Some PDSA groups consisted only of two members; teachers in these groups noted that the 
conversation was limited without the network leader to facilitate the final meeting. Almost half of 
BMTN teachers (47 percent) viewed meeting with their PDSA groups as only “moderately useful” 
or only “a little useful,” and thus did not have the ongoing, meaningful network support during the 
year that network leaders intended. 


Better Math Teaching Network: Year 2 Developmental Report | Contents | 


92 


Overall, the reduced supports in Year 2 were described by half of the returners interviewed (53 percent) 
as a reduction in collaboration: 


66 I felt like there was a lot less collaboration, a lot less people who 
were working in groups. I felt like everybody was doing their 
own individual work this year. And while that’s great, I just find it 
frustrating because you are limited by your own brain and your 
own ideas. My first idea was terrible, and it wasn't until I had 
conversations with other people that my project actually shifted 
into something that I wanted it to be. All of a sudden it became 
what I thought was a great project.... I wish there was a way to get 
more of us working together so that we're not just trying to think 
of these ideas off the top of our head and then be some kind of 
revolutionary. Because if I already had the idea, would I already 
be doing it?” 


Adapting tools, routines, and roles for scaling 


As the network grows, it is adapting the tools, routines, and roles to respond to participant needs 

in a more generative and sustainable way. Given what we have learned about variations in participant 
needs based on experience levels, shifts in the sense of connectedness, and the challenge of maintaining 
quality supports as the network grows, we offer a few considerations for ongoing design and 
implementation work: 


¢ Institutionalized mechanisms for transitioning new members 
Network leaders are building a case exemplar that provides a detailed example of 
planning and implementing a PDSA cycle. This can be made available virtually to all 
new members. Network leaders also, wherever logistically possible, put new members 
into PDSA groups with returning members. What other aspects of the work can be 
codified and institutionalized? 


* Scaling access to expertise and rigor 
Network leaders have played a critical role in teachers’ learning and engagement with 
PDSA cycles. How will expertise, rigor, and perspective be available within the network 
and to whom? 


¢ Scaling quality controls 
We know that the participation structures (e.g., network meetings, PDSA meetings) are 
the critical components for engaging members in the work of the network. As network 
leaders pull back from their direct roles in some of these structures, how might quality 
of interactions and work still be monitored and shaped? For example, PDSA groups are 
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an important space to build knowledge and support for the deep and in-between meeting 
work that teachers do, but we know that not all of these groups end up with active and 
productive engagement. Without a network leader at the table, some of these groups 
may actually dissolve. How might network leaders measure and monitor the relative health 

of PDSA groups and intervene as necessary to shape the quality of the work? 


Differentiating work and roles 

It is clear that new and veteran network members have different needs and assets. 
What models will network leaders employ to integrate new and veteran network members 
and ensure that the work of the network can be conducted with more members? In Year 3, 
network leaders have designed new leadership roles for veteran teachers. What are we 
learning about how the shifting roles do and do not meet the needs of all network members? 


Managing knowledge 

As the network grows, so too does the volume of work produced by its members. This 
creates a new challenge for members and the network overall: it becomes more difficult 
for network members to have a clear, conceptual understanding of the various strands of 
work and learnings that are emerging from individual teachers’ PDSAs. How might network 
leaders capture, curate, and disseminate promising change ideas, tasks, measurement tools, 
and other useful resources, and make them accessible to network members? How might network 


leaders create a coherent representation of what has been learned so far about the network aim? 


Implications: 


As networks expand their membership, the needs of members change. This requires 
network leaders to continuously tweak the design of supports to new and returning 
members, which might include restructuring participation structures, constructing 
new tools and resources to showcase the learning and manage new learning, and 
differentiating member experiences. 


Returning members in a growing network can be assets to the new members, but also 
must have their changing needs met. How might the network be designed to ensure 
that returning teachers have access to the expertise that will stretch their practice 

and understandings? 
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Chapter 8: Spread in Year 2 


A networked improvement community is a network of practitioners and researchers who collectively 
work on an important problem of practice. These networks measure success by tracking progress toward 
a common aim and use improvement science methods to learn how to improve practice. One promising 
feature of an instructionally focused NIC is that the field of education can potentially benefit from the 
NIC’s identification of promising instructional practices in order to accelerate improvement in other 
schools and classrooms. This chapter examines the different ways in which the Better Math Teaching 
Network worked to spread its impact in Year 2. 


Informal spread 


Naturally, teachers in the network have been excited to share what they are learning with colleagues. 
There are a number of informal mechanisms BMTN teachers are using to spread their learning to 
colleagues in their schools. In Year 2, 84 percent of BMTN teachers surveyed reported that they shared 
their work with colleagues outside of the network. The majority shared individually with other math 
teachers at their school, but some shared at their school more broadly. See Figure 30 below. 
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Figure 30: Proportion of teachers reporting various activities to spread 
the BMTN work 
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A few BMTN teachers shared with educators beyond their school context. When asked what they shared, 
most BMTN teachers who shared beyond the network focused on sharing math-related elements, but 
slightly more than half of the teachers who shared have spread the improvement science approach. 

See Figure 31 below. 


Figure 31: Proportion of teachers reporting the content of what they shared 
beyond the network 


What teachers are sharing with local colleagues 


Math ideas (n=28) 
Math routines (n=26) 


Math tasks (n=25) 


Improvement science process (n=17) 


Better Math Teaching Network: Year 2 Developmental Report | Contents | 96 


In interviews, teachers described ways in which they shared their BMTN work with colleagues outside of 
the network: 


6 When we discuss instruction [at my school], 1am focused on 
student-centered aspects. In addition, I encourage teachers to try 
out a small idea and collect data, then decide if they want to keep 
it or refine it.” 


I have offered strategies that I used in the classroom that worked 
forme ormy BTMN colleagues. As a department, we have 
collaboratively used ideas that I found worked. I also have a 
colleague in BTMN, and we have collaborated on some change 
ideas for success in our classrooms.” 


6 I encourage [my school colleagues] to try my change ideas that 
have been successful or other ideas that Iam using that I have 
learned from my network colleaques.” 


The following vignette describes how a teacher who joined the network in the pilot year took ideas from 
the pilot year and used them in a context beyond the network. 


Vignette 7: Integrating continuous improvement 
into practice 


Brianna is an experienced teacher who joined the network as a pilot teacher in 2015. She was already 
quite student centered in her practice and was involved in other professional learning opportunities as 
she sought new ways to learn with and teach others in her urban district. In her third year of the work 
with BMTN, Brianna was ready to push her learning beyond her classroom and the network. 


Brianna considered the PDSA cycles to be a good tool for improving practice. She also appreciated the 
collaborative nature of her small group in the pilot year, when a few teachers tested out the same change 
idea, collected data, and reported back to each other. She wanted to bring inquiry cycles and this idea 

of joint work into another project in which she was engaged. Brianna simplified the PDSA template and, 
with another BMTN colleague, used it to guide improvement work in a new context. She explained, “We 
were doing a similar data inquiry cycle around testing instructional techniques in the classroom, and we 
actually used a more simplified version of the PDSA form.” 
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Her simplified form had the following elements: 


* Adescription of the change idea 

¢ A Plan 

* Three questions under “What do you want to learn?” 
* Will students engage? 
* Will students engage with depth? 
* Will | do it as frequently as planned? 

* A“Do” section: Describe what happened 


e Study and Act 


By maintaining key elements of the inquiry process, but shortening the template, Brianna sought to 
simplify the effort required of teachers for documentation. Since she did not find the prediction step in 
the BMTN PDSA cycle to be useful, she left that out of her revision. 


She shared it with two colleagues in the same district, both of whom were seasoned teachers and had 
engaged in inquiry-based work prior to their collaboration. Brianna described how they used the form for 
work testing out feedback they gave to students: 


We all filled out our data in the same sheet. We were testing a feedback idea, giving students 
feedback. It was really neat to actually see the different classes. One teacher is a special education 
literacy specialist with third-graders, one teacher is a seventh-grade math teacher, and then 
[BMTN colleague] and | both teach Algebra |. It was really powerful to see a strategy work across 
classrooms. 


Brianna went on to describe that while there was variation across their four classrooms, the form allowed 
for themes to emerge that were easy to analyze. Reflecting on this work, Brianna shared, 


| like trying the same change idea with other teachers. ... | think you get more data. | think that a 
theme in your classroom might resonate with another teacher, or you both might find the same 
thing. For example, we were testing this idea of matching feedback to student work, and we all noted 
high engagement among students. Students really liked the technique, and that's something that 

| might be able to say about my own classroom for my idea, but | think when we see it as a more 
universal practice, it can be more beneficial. 


When Brianna reflected back on this work within the BMTN context, she shared: 


One of the things we are trying to do in the network is come up with strategies that people can take 
to scale. | think when you have a bunch of teachers working on the same idea, you're getting better 
data to support that idea. | think trying the same idea in a range of classrooms just adds more 
support to whatever the change idea is, if it works. 


SS 
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In Year 3, network leaders are more formally supporting BMTN teachers to share their BMTN work with 
their school or district colleagues. Several teachers have signed up to share in a variety of ways that 
include professional development sessions for district math teachers, school-based opportunities for 
math teams or—in some cases—for all content area teachers. BMTN teachers planned to teach their local 
colleagues about new math routines, inquiry cycles, and/or student-centered teaching. 


Formal spread 
In the first two years, BMTN network leaders designed four mechanisms for the network to spread its 
impact, which we examine below. 

* Creating a public face of the network via Facebook, a website, and a monthly newsletter. 


* Encouraging BMTN teachers to present to the broader math education community via 
conference presentations and blogs. 


* Establishing a parallel network of school, district, and state math leaders. 
* Establishing a learning community of new teachers to test out BMTN change ideas. 


See Figure 32. 


Figure 82: Different ways in which BMTN leaders design for spread in Year 2 
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The public face of BMTN 


Starting in Year 1 and building in Year 2, the network’s communications team built the public face of 
the network in three venues: 


1. Website. The network launched its website early in the network's development. Over time, 
network leaders added content to the site. In addition to sharing information about the 
network, the website serves as a repository of BMTN member-generated ideas for change. 
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2. Facebook page. The network launched a Facebook page early in the network's development. 
This is a place where anyone can go to learn more about the network and where teachers 
can link ideas, invite others to conferences, and connect in other ways. 


3. Monthly newsletter. In October 2017, the BMTN hub launched the first monthly BMTN 
newsletter. Sent to all members, and anyone who has subscribed on the website (currently 
150 subscribers and counting), this newsletter shares member profiles, news of upcoming 
events and articles of interest, and news of the network. 


BMTN member presentations and blogs 


The Hub encouraged BMTN teachers to present at local and national conferences and to write blogs. 
Ten teachers (25 percent) presented or will present at conferences and one teacher has blogged about 
her work in BMTN. Teachers have presented at the National Council of Teachers of Mathematics 
Conference as well as at state and regional conferences. They have presented on a wide range of 
topics including how they use improvement science to improve instruction, how they have developed 
and implemented new routines in the classroom, and how they define deep engagement in algebra. 
Additionally, some members presented on the work within their schools or districts. 


Leader network 


In the pilot year of the Better Math Teaching Network, 11 teachers were joined by three district and 

state leaders who engaged side-by-side with the teachers in the network meetings and then—to varying 
degrees—took the ideas back to their local contexts. One of these leaders moved into a Hub role when 
the network launched the following year. In the first full year of implementation (Year 1), network leaders 
continued to work with school, district, and state leaders, but in a different model. They held two virtual 
meetings to keep leaders in the loop and bring new leaders into the work. Leaders were invited to 

the year-end full-network meeting to hear what BMTN teachers were sharing and to celebrate BMTN 
teachers’ learning. 


In Year 2, network leaders again designed a parallel meeting structure for the leader group comprised 
of two in-person meetings in fall and late winter. 


* September 2017: Five district leaders and three school leaders attended the kick-off meeting. 
Two state leaders were interested, but unable to attend. Participants learned more about the 
Better Math Teaching Network, improvement science methodology, and some of the change 
ideas that BMTN teachers were trying in their classrooms. Network leaders asked leaders to 
try out a BMTN teacher's change idea in their own context. 


* December 2017: Two state leaders, one district leader, and two school leaders met to continue 
their learning, share their experiences from trying out the BMTN change idea with teachers in 
their local context, and plan for future work. 
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Network leaders refocused the leadership work after the second meeting. They sought to build 
professional learning communities in local contexts as a way to spread the improvement science 
approach and continue to try out BMTN teachers’ change ideas in new contexts. A state and school leader 
showed interest in continuing the work in this way, so a network leader partnered with them to grow the 
work in their local context. See BMTN Spread: Building a District-Based Learning Community. Members of 
the network leader group were also invited to the end of year BMTN network meeting in which teachers 
shared their learning from the year. Two state leaders attended this meeting. The other six leaders did 
not continue with BMTN. 


The continuous improvement element of the BMTN strongly resonated with several of the school, district, 
and state leaders, particularly with those who engaged more deeply in the pilot year. There is evidence 
that some leaders have integrated BMTN ideas into their own work and/or to support the learning, 
growth, and improvement work of others they lead. 


A district I’m working with needed a way to think about how they 
are going to identify how they want to change, and how they want 
to pick data. Some of the tools [from BMTN] were a good segue 
into that work—they were a good way to talk through that.” 


66 I brought the whole concept of the driver diagram in articulating 
your goals and picking one thing to focus on to my work here at 
the [state] department—pushing us as a team to think about: what 
are the things we need to accomplish? Which is going to give us 
our biggest bang for the buck?” 


6 The teachers who are going to become math coaches go through 
this training, but we also have assigned field coaches from the 
university that go out and observe them in their teaching and 
provide actionable feedback and see how they are working as a 
coach. We've used this work to think about: What kind of feedback? 
How is your interaction with the teacher youre coaching? What 
strategies do you need to improve on? We are using that cycle 
with our field coaches to support the coaching candidates with the 
classroom teachers that they are working with. I've kind of woven 
it into a lot of things I do.” 


Some leaders also appreciated the wide view of high school math teaching it provided for them and the 
challenges with which teachers were grappling. 
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66 I have an opportunity to hear from teachers across the New 
England states about what they are trying, what's working, what's 
not working. It gives me insight into areas of need for professional 
learning so that when I think about my next steps at state level, 
what am I going to provide for support sent to teachers out in 
the field?” 


Some of the state, district, and school leaders had ideas about how the network might further use their 
expertise. For example, one leader suggested that participating leaders might: 


* build a database of resources and supports for teachers to access so they are not spending 
time looking for good materials. 


* provide training around research-based strategies to support teachers in their PDSA work. 
“What does that strategy look like or feel like as a student?” And then flip it around and say, 
“What are the teacher moves behind that?” so that the students have that value-added 
experience with that routine or strategy that the teachers are trying to implement in their 
change process. 


Building a Learning Community 


In February of Year 2, BMTN network leaders launched a learning community at a New England high 
school. The math department chair of this high school, who had attended the meetings for state, district, 
and school leaders, was excited to bring the ideas from BMTN to her school. Network leaders worked 
closely with her to design a model that worked within the time and resources they had available, co- 
constructing the model with her to fit her interests. The learning community was designed to: 


1. Spread change ideas and continue to test these change ideas 
2. Spread improvement science methods 
3. Pilot the use of PLCs to spread the work 


Eventually, network leaders may support these groups to identify and test their own change ideas. If 
successful, this learning community could serve as a model that BMTN teachers might take back to their 
schools and districts. 


The district-level learning community that was piloted in one state in Year 2 engaged math teachers from 
that district's one high school and both of its feeder middle schools. This was the first time that most of 
the teachers had the chance to collaborate with district colleagues and to learn math-specific content 
together. Teachers were universally happy to be given a routine to try out in their classroom and a rubric 
to assess student work. All of the teachers tried out the routines in their classrooms. There were varying 
degrees of data collection, analysis, and documentation completed. Much of the conversation focused on 
challenges with students (e.g., it was new for many of the teachers to engage their students in non-rote 
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tasks so there was a learning curve for both them and their students), challenges with finding good tasks, 
and building coherence between middle school and high school math in the district. (Please see the case 
write-up in Chapter 9 for a fuller explication.) 


Implications 


¢ There are a variety of ways in which ideas and new learning from NICs can spread beyond 
network members. Network leaders must intentionally design and manage these mechanisms, 
which in turn will focus both the ideas that are shared (e.g., improvement science strategies, 
classroom routines) and the audience for whom they are intended. 


¢ An important job of network leaders is to consolidate learning, package it into deliverable 
components, and test out successful strategies for spread. 
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Mary Lou had been the head of the math department at a high school in New England for nine years. 
When she learned about the Better Math Teaching Network, she was very interested because she 
immediately saw how the network could support her goals. 


I wanted to join the network because I was looking for some 
national help or support for my school. I have a high failure rate in 
Algebra I. I have managed to decrease that over the years—it was 
high when I started—but that was one of my biggest concerns as 


a department chair: How do I reduce the Algebra I failure rate in 
my school?” 


Along with a math leader from her state department of education's office, Mary Lou attended two 
meetings in late 2017 to learn more about BMTN. In these meetings, AIR-based hub leaders met with 
school, district, and state math leaders not affiliated with BMTN to share ideas from the teachers’ work 
and support the spread of those ideas beyond the network teachers. In the first meeting of the leader 


network, Mary Lou learned more about what the teachers in BMTN were doing, received an improvement 


science tutorial, and was given a classroom routine to bring back to her school that had been created 
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by a BMTN teacher: Inserting Non-Rote Problems into Instruction. As she and her colleagues tried out the 
routine, they found it to be useful in engaging more students. 


Mary Lou and her state leader were interested in integrating BMTN’s work in their local contexts as they 
sought ways to improve math teaching and learning. Mary Lou was particularly eager to engage her 

algebra teachers in the process of improvement cycles. Since her district is small, and there is no district 
math coordinator, Mary Lou often looks for ways to support math learning across the district. When the 


leader network met again in December, Mary Lou was ready to push the work more deeply into her local 


context. After that meeting an AIR-based hub leader, Gail, collaborated with Mary Lou to build a math 


learning community in Mary Lou's district. Mary Lou invited the math teachers from her high school’s two 


feeder middle schools to join the work as well. 


Figure 33: The timeline of the district-based learning community 


SY-Je)04D\-1e40NWa ° Leader network meetings 


* Department Chair, Math State Department of Education Leader, BMTN 


January 2018 Network Leader collaborate to start a district-based learning community 


* Three district-based meetings include algebra teachers from high school 


; and both middle schools in the district 
Se) diarewACh Ks) 


* Teachers try out two new BMTN-created routines in their classrooms in 
between meetings 


The structure of Year One of the district-based 
learning community 


Mary Lou shared the idea of the learning community with her principal and superintendent, who were 
interested. She then brought together teachers from her school and the district's middle schools for 
three meetings. The first meeting was held in February after school. In April and May, the teachers took 
a full day out of school (the network paid for substitute teachers). On those two days, Mary Lou worked 
with the teachers on district math projects in the morning and Gail joined in the afternoon to engage 
the teachers in the BMTN work. The state math leader joined the meetings and learned alongside the 
teachers. In all, nine teachers joined the learning community—four high school and five middle school 
math teachers. 
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Gail worked with teachers in the first meeting to develop an understanding of: 


¢ The problem that BMTN is tackling 
¢ The aim that guides the improvement work 
¢ The improvement science approach, and specifically the PDSA inquiry cycle 


* How to use PDSA testing to try out some routines produced by BMTN teachers 


At the end of the meeting, Gail asked the teachers to go back to their classrooms and engage in a PDSA 
cycle using a BMTN teacher's routine: Non-Rote Problems. The BMTN teacher, Ellen, built a formative 
assessment problem-solving routine to provide opportunities for students to work on challenging non- 
rote problems. In between the first and second meetings, teachers tried out a PDSA cycle using the new 
routine in their classrooms. 


When the group met again in April for their second meeting, Gail spent time engaging teachers more 
deeply in the PDSA process, helping the teachers to think about: 


* How does the PDSA process help you make decisions about the routine you are testing? 
¢ What does your data tell you? 
¢ What do you observe in terms of quality and depth? 


¢ What modifications (if any) should you make? 


Teachers brought data and student work to share with their colleagues in the second meeting. As they 
looked at their student work, they grappled with how to define depth. They did not yet have the rubric 
that Ellen had refined for use in the routine. This allowed the teachers to have deep conversations about 
what depth meant for them as they collaboratively looked at their student work. 


In addition, the teachers realized that—as a district—they did not have a set of problem-solving strategies 
that students used throughout middle school and high school. They discussed ways in which teachers 
could model strategies for students on a more regular basis, connecting problem-solving strategies 

with the tasks they were working on in their PDSA cycles. After the meeting, Mary Lou sent the teachers 
pictures of the posters she used when teaching problem-solving strategies to her students. Mary Lou and 
the teachers agreed to focus on problem-solving strategies across the math curriculum the following year. 


At the end of the second meeting, teachers were introduced to a new routine, also designed by a BMTN 
teacher: Using Written Examples to Help Students Explain Thinking. The hub leader gave teachers the choice 
to either continue to test out the first routine in their classroom or try the new one. She also gave them 
the rubric that Ellen had designed and tested as part of her Non-Rote Problem routine. The rubric is a tool 
for students to assess their own depth of engagement. Gail asked teachers to fill out another PDSA form 
as they engaged in their inquiry cycle. When they met in June, Gail continued to help them think about 
how to use the PDSA process to make decisions about the routine they were testing, focusing on the data 
they collected and what it told them. 
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Throughout the meetings, Gail balanced new learning with sharing and discussion. Teachers engaged in 
the discussions and shared their work and experiences with each other. In addition to discussions about 
their PDSA cycles, there were lively discussions about their students’ learning needs, challenges they 
faced in their teaching, what tasks they were using in their classrooms, and how to build coherence in 
their district. 


By design, the district learning community had less training on improvement science compared with 
what network leaders provided to the BMTN teachers. For some, what they learned about improvement 
science in the first meeting was enough. A few noted they would have liked to learn more. Even so, some 
teachers seemed to “get” enough of the essence. 


You guys have a great process. The PDSA seems to be a really 
effective cycle in the way of doing things... It kind of pinpointed for 
me that you do the DSA part really well, but the planning part is 
something I definitely need to work on. I've always kind of been a 
fly-by-night kind of guy. Im good like that in the sense that I can 
come up with a lesson plan quickly or I can create an assessment 
very well, in my opinion, on my own without having other 
supplements or other people helping me out. But at the same time, 
I definitely need to put more time into the planning piece.” 


6 I think is pretty self-explanatory, the concept. You know, the idea 
behind it is to do what you're doing. It’s to collect data, to interpret 
data. It’s to improve on concepts and ideas and procedures that 
are the foundation for how we are teaching and how we're doing 
things. So, I kind of—I get it in concept.” 


In addition, network leaders intentionally designed the learning community so that teachers would try out 
an already tested and refined routine, with the option to use a rubric that was designed and refined for 
that routine. This allowed teachers to: 


¢ skip the step of deciding what to change in their own practice 


* have access to a practical measure to assess whether the change led to 
an improvement in student engagement 


* engage in joint work with colleagues implementing the same routine 
* focus on testing, analyzing data, and making decisions about next steps 


* compare experiences across different classrooms and schools 
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It also provided an opportunity to spread the routine, another goal of the pilot. A few teachers mentioned 
the adjustment to their own teaching that was necessary in order to implement someone else's routine 
into their practice, but even those teachers were excited to engage students in these different, more 
student-centered approaches. 


It was nice to have a structured routine that was a step-by-step 
on what they had to do, how they had to do it exactly, and then 
[getting] feedback and rubrics to go along with the task.” 


6S I think it provides a lot more structure to be given a model. I 
wouldn't have learned how important it is to teach the routine 
before actually using it if I did it on my own, so I see a really big 
benefit to be given the routine. I feel the benefit of working on the 
same routine so we could all reflect together.” 


For me that was helpful that we were all doing the same thing, so 
you could really compare the data. If I just had one group, I think 
the data would be skewed because I don’t have the type of kid that 
somebody else has.” 


Both the middle school and the high school teachers appreciated the opportunity to work cross-level. 


6& Then we [the high school teachers] can get a better feel, too, for 
where our students are coming from, or then they can see what's 
expected of them, and where theyre going. I just really like it 
because theyre in a different environment than we are, so things 
are happening in a different way. I like to compare them.” 


Better Math Teaching Network: Year 2 Developmental Report | Contents | 108 


Benefits of engaging in the learning community 


In general, teacher responses to their experience in the learning community were overwhelmingly 
positive. In interviews, they noted the following benefits: 


* Opportunity to collaborate with other math teachers in the district, including teachers 
from another level (middle school and high school) 


6 It was useful because we got to talk to the other math teachers, 
which we don’t usually get to... like the high school teachers and 
even the other middle school teachers. And we talked about the 
problems that we did, because a lot of us did the same problems.” 


I find it really useful when algebra teachers or geometry teachers 
can really have that time to collaborate and share new strategies 
that might work for them or their population.” 


66 The focus was very concise, and we were happy that we are able 
to all get together and discuss such important things. Because 
really, aside from the administrators in our buildings, we dont 
necessarily discuss our teaching strategies as much as resources 
or the day-to-day tasks with our peers.” 


¢ Accountability and investment 


66 This [opportunity] is different because the teachers have to do 
something and come back and report on it. There is investment. If 
they didn’t want to do it, then they wouldn't have come back with 
work. They would have just said, “I can’t come.” The teachers that 
you saw, it was evident that they are invested in it. They came 
back with work. They were able to discuss the work. And to me, 
that is ownership and interest and motivation.” 
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* Opportunity for novice teachers to learn from more experienced teachers 


For me it’s really helpful to hear the perspective of more 
experienced teachers. I talk about what I've learned and what 
happened, and they can help me get a different perspective on it 
that I wouldn't know yet because I’m a new teacher.” 


Challenges teachers faced in the learning community work 


Teachers in the learning community faced some of the typical challenges related to this work, including 
the difficulty of getting pulled out of class to meet, finding the extra time it takes to do the work, and 
starting new routines midyear. 


There were a lot of interruptions where if you lay out the scope 
and sequence and the plans that we had laid out throughout the 
year for pacing and fluency. It was tough to kind of fit in the time, 
albeit necessary time, for students to engage in a process like this. 
It’s different than what we are used to. So that... was the biggest 
challenge, trying to get this in and make time for it with everything 
that was going on.” 


I don't Rnow how other people feel about professional development, 
but I do not like to leave my classroom. Because then when I get 
back, nothing was done that I wanted to be done. And then I’m 
stepping—Im taking a step back, because I have to review with 
them. I don't like that.” 


Some teachers mentioned the time it took to fill out the PDSA documentation. In fact, very few teachers 
turned in their PDSA forms, despite this being an expectation set by Gail. Some teachers did not know 
how to fill out the form, while others had technical trouble (e.g., not knowing how to access it digitally, 
unable to figure out how to send it to Gail). 


What teachers learned by engaging in the learning 

community work 

Teachers reported learning a variety of things from their engagement in the learning community. Several 
teachers appreciated that the community helped them to feel like they were not alone in trying to get 


their students to engage in math. For three teachers, the work helped them think differently about how to 
use data to improve their instruction. 
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66 T really think I've learned how to read data or how to interpret 
that data. Instead of taking a student and saying, “Okay. They 
know this strategy. I’m going to put them in this pile. These kids 
don’t know and I’m going have to go over it again,” Im not doing 
that. I’m looking at actually what they did and saying, “Okay. I’m 
going to pair them up with this student who got it right and then 
let them talk together to come up with a compromise of how they 
both could solve the question.” 


I’mvery good at looking at outside measures of data, 
standardized tests. But looking at your own data, that was a new 
approach to me.” 


I think those skills; how to make something better—that's the 
theme that I get from Better Math Teaching Network. If you want 
to make something better, then focus on it and just collect the data 
and define it. Figure out what's wrong.” 


Another important learning this work brought to several teachers (and a reminder in the case of one 
veteran teacher) was the need to do more complex tasks. 


Trying to fit in these quality tasks that allow students to dig deeper 
into the concepts and reach for tools that they've either learned 

or might be learning that—it needs to just be more of a part of the 
weekly process in the classroom.” 


As they tried out and discussed their use of similar tasks, teachers identified tasks that worked better at 
the high school level or middle school level. 
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Conclusion 


AIR-based hub network leaders designed the learning community pilot as a way to test a model for 
spread: might teachers be able to accelerate their learning by taking up routines developed and tested 
in other contexts? Could they productively engage in inquiry cycles with less upfront training 

in improvement science and less ongoing support than the full BMTN network model? In this pilot: 


* The learning community engaged math teachers from the district’s high school 
and middle schools. 


* Teachers ranged in experience from first year to veteran status. 


* This was the first time most of the teachers had collaborated with district colleagues 
who did not teach in their own school. 


* The math leader utilized the opportunity to have all of the teachers together to work 
on district matters prior to the start of two of the three learning community meetings. 


¢ All of the teachers tried out at least one routine in their classrooms. 


* There were varying degrees of data collection, analysis, and documentation completed. 


There were two main trade-offs of this design compared with the BMTN model: all teachers engaged 

in the same routine rather than designing their own work and these teachers learned less about 
improvement science than the BMTN teachers. While it is too early in the work to know definitively, early 
evidence points to the payoff in simplifying the BMTN message. Teachers were universally happy to be 
given a routine to try out in their classroom and a rubric to use to assess student work. They appreciated 
the joint work and opportunity to dig into the student work and measurement piece. They spent less time 
both generating a change idea and grappling with what makes a good practical measure. In this first year, 
much of their discussion focused on: 


* challenges with students (e.g., it was new for many of the teachers to engage their 
students in non-rote tasks so there was a learning curve for them and their students), 


* challenges with finding good tasks, and 


* building coherence between middle school and high school math in the district. 


As the learning community enters its second year, all but one of the teachers from Year 1 are returning 
and six additional teachers (from each of the three schools) will join. Mary Lou has set two goals for the 
work in Year 2: to continue to focus on problem solving across the grades and to increase SAT/PSAT 
scores. When the math team at the high school looked at the item analysis of their students’ SAT scores, 
they noticed their students struggled the most with word problems. Mary Lou will use the learning 
community as a place for teachers to collaboratively work on this challenge as they provide more 
opportunities for students to solve with quality and persevere through hard problems. The teachers new 
to the group will kick off their work in December, and the returning teachers will join for three meetings 
in January, February, and March. We are interested to learn more about how this work spreads and 
deepens, changing teaching practice and student engagement in math in grades 6-12. 
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Conclusion 


The Better Math Teaching Network seeks to increase opportunities for high school students to deeply 
engage in algebra. Teachers in the network are highly engaged; they spend time outside of their typical 
teaching duties to attend quarterly network meetings and engage in quarterly inquiry cycles supported by 
virtual small-group meetings. Some BMTN teachers participate in optional study group meetings, support 
the network through leadership roles, and spread what they are learning beyond the network. Seeking 

to improve their own practice, BMTN teachers learn from each other and from network leaders. As the 
network moves into its third full year, it is poised to consolidate learning from several years of PDSA 
testing, build structures to spread that learning beyond network members into their schools, districts, 

and new districts, and inform the field on what it takes to grow and sustain an instructionally-focused 
networked improvement community. 


| Contents | 113 


Glossary 


BMTN: Better Math Teaching Network 

The Better Math Teaching Network is a networked improvement community (NIC) focused on student- 
centered learning as a potential solution to the problem of high rates of high school math students 
disengaged in mathematics learning. The network focuses on algebra content. 


Change idea 

Change idea is a term from improvement science that is defined as: “An alteration to a system or process 
that is to be tested through a PDSA cycle to examine its efficacy in improving some driver in the working 
theory of improvement,” (Bryk et al., 2015). In the context of the BMTN, a change idea is a small change 
to planning or instruction that a teacher makes in order to examine its efficacy in improving deep student 
engagement in algebra. 


DEA: Deep Engagement in Algebra 
The Better Math Teaching Network defines deep engagement in algebra as the extent to which 
students are: 


¢ Making connections among mathematical procedures, concepts, and application to 
real-world contexts, where appropriate (connect) 


¢ Making sense of and solving challenging problems that extend beyond rote application 
of procedures (solve) 


* Communicating and justifying their mathematical thinking as well as critiquing the 
reasoning of others (justify) 


These three elements of deep engagement in algebra (connect, solve, and justify) are referred to by the 
BMTN as DEAs. 


Driver diagram 
A tool used in improvement science to represent the theory a group has for improvement in a particular 
problem of practice. The driver diagram guides the improvement work, providing shared language. 


Hub 

The Hub is the name of the team that leads a networked improvement community. The Hub is 
responsible for technical support for the NIC as well as for organizing the joint work of the NIC and 
articulating learning within and beyond the network. In the case of BMTN, the Hub consists of AIR 
researchers who were former math teachers. 


Improvement science 

Improvement science is an applied science that has dramatically improved practice in a number of 
industries by helping practitioners learn their way into improvement. The approach has a long history 
in the manufacturing industry and subsequently the healthcare field and provides a disciplined 
methodology for learning from practice to improve the systems and processes that shape work within 
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organizations (Berwick, 2008; Deming, 2000; Gawande, 2007; Langley et al., 2009). More recently, 
education reformers and leaders have looked to improvement science as a way to accelerate large-scale 
improvement in schools and districts (Lewis, 2015). 


NIC: Networked Improvement Community 

Networked improvement communities, or NICs for short, bring together practitioners, educational 
leaders, and researchers in order to solve a pressing problem of practice (Bryk, Gomez, Grunow & 
LeMahieu, 2015; Hannan, Russell, Park & Takahashi, 2015). In promoting the use of improvement science 
in networked communities, Tony Bryk and colleagues at the Carnegie Foundation for the Advancement 
of Teaching have recently promoted the NIC concept as a way for practitioners to learn how to improve 
education at scale by building an evidence base about both productive practices and knowledge of 
implementation processes to address persistent problems of practice and policy (Bryk et al., 2013). NICs 
are professional learning communities distinguished by four essential characteristics: (1) they are focused 
on a well-specified common aim; (2) they are guided by a deep understanding of the problem, the system 
that produces it, and a shared working theory of how to improve it; (3) their work is disciplined by the 
rigor of improvement research; and (4) they are coordinated to accelerate the development, testing and 
refinement of interventions, their rapid diffusion out into the field, and their effective integration into 
varied educational contexts (Bryk et al., 2015; Russell, et al., 2017). 


PDSA: Plan-Do-Study-Act Cycle 

The PDSA cycle is an inquiry routine involving four steps: Plan-Do-Study-Act. The logic of the cycle is 

that teachers learn how to improve their practice by planning a specific change tied to a working theory 
of improvement, testing the change, studying evidence to assess whether the change constituted an 
improvement, and deciding what action to take in light of what was learned. Identifying hypotheses, 
testing those hypotheses, and comparing results with one’s predictions generates new details about 
one’s practice. This rapid inquiry generates new learning and allows teachers the opportunity to examine 
variations in context that support or constrain their practice. 


Process map 

A tool used in improvement science that is defined as: “A tool for visualizing the steps in a process that 
can assist an improvement team in identifying gaps, strengths, and opportunities for improvement,” (Bryk 
et al., 2015). In the context of the BMTN, teachers built process maps of a teaching routine in order to 
identify places in their practice routine to focus their improvement efforts. 


Small group PDSA coaching meetings 

In order to support teacher engagement and promote social learning, the network organized virtual 
meetings of two to three teachers plus a network leader focused on planning and debriefing inquiry 
cycles. These monthly meetings provided a forum for teachers to identify aspects of their teaching 
practice to target for improvement, generate potential change ideas to test, and discuss the results of 
these inquiry cycles. In these meetings, teachers had the opportunity to dig into specific instructional 
challenges and improvement strategies with teachers teaching the same (Algebra |) content and a Hub 
lead with research and practical expertise in mathematics teaching and learning as well as improvement 
science expertise. 
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Appendix A: Evaluation Methodology 


While some industries, such as healthcare, have used improvement science for decades, the use of 
improvement science and networked improvement communities is new to education. Because this work 
is complex, innovative, and relatively new, and because improvement science by nature requires rapid 
tests of change, adaptation to context, and systems thinking, we utilized a developmental evaluation 
approach, rather than formative or summative evaluation, to understand and inform the Better Math 
Teaching Network. 


What is developmental evaluation? 


The purpose of a developmental evaluation is to support the development of innovation and adaptation 
in dynamic and complex environments. Indeed, developmental evaluation is best suited for situations 

of high complexity and when working on early stage social innovations, like the Better Math Teaching 
Network (Gamble, 2008). The team that is involved in a developmental evaluation acts as an insider- 
outsider, serving as a partner in the iterative design of the NIC, while also reporting out findings with 
sufficient frequency to improve learning and support design decisions. Developmental evaluation 
incorporates methodological flexibility, systems thinking, and adaptability. The ability to facilitate rigorous 
evidence-based perspectives, while tolerating ambiguity and agility, allows a developmental evaluation 
team to evaluate the complexity of the work of an evolving NIC (Patton, 2006; 2010). Since the NIC 
concept is too new to be able to clearly identify if a network is “implementing with fidelity” or getting to 
a“prescribed outcome,” drawing on principles of developmental evaluation will inform the work with 
structured, real time analysis intended to refine the network, push the learning, continue to build capacity 
within the team, and inform all stakeholders of progress. 


Developmental evaluation emerged in response to the need to support real-time learning in complex 
and emergent situations. Traditional forms of evaluation work well in situations where the progression 
from problem to solution can be laid out in a relatively clear sequence of steps (Gamble, 2008). However, 
initiatives with multiple stakeholders, high levels of innovation, fast paced decision-making, and areas of 
uncertainty require more flexible approaches (Patton, 2008). 


Developmental evaluation differs from traditional forms of evaluation in several key ways: 


* The primary focus is on adaptive learning rather than accountability to an external authority. 
* The purpose is to provide real-time feedback and generate learnings to inform development. 
* The evaluator is embedded in the initiative as a member of the team. 


* The evaluator role extends well beyond data collection and analysis; the evaluator actively 
intervenes to shape the course of development, helping to inform decision-making and 
facilitate learning. 


* The evaluation is designed to capture system dynamics and surface innovative strategies 
and ideas. 
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¢ The approach is flexible, with new measures and monitoring mechanisms evolving as 
understanding of the situation deepens and the initiative’s goals emerge (adapted from 
Westley, Zimmerman & Patton, 2006). 


Michael Quinn Patton (2008), who pioneered this form of evaluation, defines it this way: 


Developmental evaluation refers to long-term, partnering relationships between evaluators and 
those engaged in innovative initiatives and development. Developmental evaluation processes 
include asking evaluative questions and gathering information to provide feedback and support 
emergent decision-making and course corrections. The evaluator is part of a team whose 
members collaborate to conceptualize, design, and test new approaches in a long-term, on- 
going process of continuous improvement, adaptation, and intentional change. The evaluator's 
primary function in the team is to elucidate team discussions with evaluative questions, data 
and logic, and to facilitate data-based assessments and decision-making in the unfolding and 
developmental processes of innovation. 


Developmental evaluation is suited to situations that are: 


* Highly emergent and volatile (e.g., the environment is always changing) 
* Difficult to plan or predict because the variables are interdependent and non-linear 


* Socially complex, requiring collaboration among stakeholders from different organizations, 
systems, and/or sectors 


* Innovative, requiring real-time learning and development (Patton, 2008; Gamble, 2008) 
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Data collection 


Data collection was cued to the network design and activities. The table below summarizes the data 
sources and indicates the number of administrations and respondents. 


Table 1A: Data sources and number of administrations and respondents 


Data sources Data collected 

Network leader self-assessment survey N=3 May administration to all network leaders 
Network member survey 41 teachers December; 38 teachers June 

Teacher interviews 40 teachers December; 38 teachers June 

Observation and artifacts from network in-person Field notes and artifacts: July, October, December, May 
meetings 

Small group, PDSA coaching meetings videos N=47 


PDSA documentation 


Cycle 1 drafts N=36 
Cycle 2 drafts N=35 
Cycle 3 drafts N=33 
Cycle 4 drafts N=26 
Change idea summaries N=34 
Change idea summary presentations N=29 
Study group meeting videos N= 18 (video from 3 groups) 
Study group meeting notes N=38 (notes from 7 groups) 


BMTN case studies 


Classroom observations N=8 
Case study teacher interviews (BMTN) N=16 
Case study teacher additional artifact documentation 24 artifacts of change in practice 


LC Case study 


Meeting observations Field notes and artifacts: February, May 
Artifacts: April 


Case study teacher interviews (LC) N=6 


Case study leader interviews N=2 
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Analytic approach 


Survey data of members were analyzed in two ways. First, descriptive statistics were used to understand 
group level responses. Second, some survey items were imported into ORA software package and social 
network analyses were conducted. These analyses allowed us to explore and visualize the variables 
shaping formal and informal relationships within the network. 


PDSA documentation were analyzed using a rubric to assess the quality and completion levels of key 
aspects of the PDSA process. 


Field notes and artifacts from network meetings, school-based meetings, and coaching meetings 
were analyzed using thematic analysis tethered to our guiding evaluation questions. 


Collectively, the entire data set was analyzed utilizing grounded theory to discern key aspects of network 
development that will ultimately inform our understanding of how NICs develop and mature. 
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